:W&Bá¾}ÏäWM 1d6qz63uª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0عÙt‚‚" qvxrbjuuggim 1d6qz63uŠÞ 1d6qz63uescheuller-uc-san-diego spatop-sweep" _wandb‚{}  batch_size‚2048  dropout‚0.23244339818615536  hidden_dim‚128 & learning_rate‚0.000434999776036891  num_embedding_layers‚6Bdpfx_vanilla_1d6qz63ubnqupkqclj%dp-spatop-sweepagent-vanilla-v2-h9cgsŠ ƒÅìÒ˜±ï¥š3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64¢úÓb; 1d6qz63u*  _wandb‚ {"runtime":0}  _runtime‚05TcaÒÛ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 ƒÅìÒ˜ÝÙb*-of*>/data/spatop/options_files/sweep_fixed_dp/config_1d6qz63u.json*-l* /scratch/logs*-n*dpfx_vanilla_1d6qz63u2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº ach73iijbx8qmow9e3rc0i3xeuptpv79202õÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ÌÁ¦ª€àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º ach73iijbx8qmow9e3rc0i3xeuptpv79µª½Ç2  requirements.txtö—îW 1d6qz63uZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64WhY 1d6qz63uZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64Mlt]*  1d6qz63ub trainer/global_step2jmæô\  1d6qz63uZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64(ɸ?/  1d6qz63ub*"trainer/global_step2Ê5äñJ  1d6qz63uj9 „ÅìÒ˜äÊ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ¯;í?’  1d6qz63uj€  „ÅìÒÀEî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¶@6‘[ 1d6qz63ujÉ „ÅìÒ˜¼»F·Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.814 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 >k²p 1d6qz63uj_ „ÅìÒÍÒFPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). •Ò,z 1d6qz63uji „ÅìÒàÄÚFZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). È{fp 1d6qz63uj_ „ÅìÒ âåFPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ÿ:ës 1d6qz63ujb „ÅìÒÀéíFSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). }µRm 1d6qz63uj\ „ÅìÒ¨ÜôFMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). —Š ¤ð 1d6qz63u*Þ  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"e0·ê 1d6qz63ujØ „ÅìÒ ÆJÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. «{`M 1d6qz63uj» „ÅìÒÎœJ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. èõ ë 1d6qz63ujÙ †ÅìÒÐ…Ï¿È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. oÁºëP 1d6qz63uj¾ †ÅìÒ‘à¿­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ?1X´ž‚(:” ’ÅìÒ÷×ð¼ gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.842600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`örat‚(:ê ’ÅìÒÿ±À disk.dm-1.in‚0 proc.cpu.threads‚17& proc.memory.availableMB‚ 603.546875 network.recv‚42040 disk.dm-0.in‚0 disk.dm-0.out‚ 2.873046875 disk.sda.out‚0.53125 disk.sda3.in‚0* proc.memory.percent‚16.105127334594727) disk./.usagePercent‚39.30197687174687 disk.dm-1.out‚0.53125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚96.31624221801758 cpu‚26.270144352396738 disk.drbd2165.in‚0 disk.rbd0.in‚0 network.sent‚32822 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.sda3.out‚0.53125 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚0" disk.nvme0c0n1.out‚ 2.873046875 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 2.873046875 disk.sda.in‚0 disk.rbd0.out‚0# disk./.usageGB‚585.627857208252 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2638.6640625Ú:æ8ž‚(:” ¡ÅìÒò”²º gpu.0.gpu‚44.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þb¼&‚(:ƒ ¡ÅìÒ׸¹¼ disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚0.0625$ disk.nvme0c0n1.out‚ 7.69775390625 disk.nvme0n1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 0.79296875 network.sent‚41441 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0 disk.sda.out‚ 0.79296875 cpu‚26.217984957010096 disk.sda1.in‚0 disk.dm-0.out‚ 7.69775390625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2638.69140625 network.recv‚45916 disk.rbd0.in‚0$ memory_percent‚96.31798267364502* proc.memory.percent‚16.105294227600098( proc.memory.availableMB‚ 603.26171875* disk./.usagePercent‚39.301984295965354 disk.dm-1.in‚0 disk.dm-1.out‚ 0.79296875" disk.nvme0n1.out‚ 7.69775390625 disk.sda3.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚585.6279678344727+vjÃ3b/ lr-AdamW/pg1"trainer/global_step2:JP#©Îù3b/ lr-AdamW/pg2"trainer/global_step2:JPK=ø…-) _wandbruntime‚43  _runtime‚43 Q%Ñ ½ trainer/global_step‚49 % _timestamp‚1.7843575517178066e+09 _runtime‚ 43.793169909 _step‚0 ' lr-AdamW/pg1‚5.3287472564519146e-05 ' lr-AdamW/pg2‚5.3287472564519146e-05ÜoûQÄ!¿ % _timestamp‚1.7843575517178066e+09 _runtime‚ 43.793169909 _step‚0 ' lr-AdamW/pg1‚5.3287472564519146e-05 ' lr-AdamW/pg2‚5.3287472564519146e-05 trainer/global_step‚49/2 ði"be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPg÷¯4j#bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP¬’n$bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP¨ÅÌâp%bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP+Dœƒn&bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPÉðox'bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP€ëÞ·x(bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP¤åý‡t)bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPCùgÔk*bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPécLn+bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP |Ø‚W,bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPÐÛjh-bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP¬gB"n.bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP½z0o/bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP#¨*Ùf0bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP¢ïÕc1b_ #i>be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPÔ™“i?be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPä«Dfi@be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPëN%oAbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPÄ¡+ÚiBbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP7ìçfCbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPÂÙöElDbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP §=HkEbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPfgägFbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP$_ùCcGb_ 6á_Ob[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP¶Ã0VlPbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPŸ–YpQbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPVh}`Rb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP"@p²hSbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP¾´`áiTbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPr]lUbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPûB:ScVb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPÓ>ypkbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP{lLjlbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPlP_smbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPQ¤É0rnbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPoß.ónobj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPú‹Ö¬qpbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP'–ïûmqbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP£’œ`rb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPtË~¥qsbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP Ž÷otbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP–\g'gubc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPB[)fvbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPç#™×nwbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP¶ýB½mxbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPHâ<_yb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPÂúfzbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP‹?™j{bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPPÁhs|bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP6rl}bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPŒm•µj~bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPé­»sbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP¨æ8n€bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP‡`ülbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP³Aeæt‚bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPEqÐcƒb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP ¯n„bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPÏWà¾f…ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPÌASe†b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP´<7,r‡bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP«z}=nˆbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP LIh‰bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP©UHiŠbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP|ïët‹bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP Z¯hhŒbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPZçbb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPv§MéfŽba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP4¢|Wfba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPÜ5­µobj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP7ÿ²‘s‘bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP3TfÑb’b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP˜ÒÁj“be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPyÐBéq”bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPó–‚îf•ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPÐRFÍg–bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPDiã’k—bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPÍ»ôk˜bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP©¬e™b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPS)Áhšbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPÞo›bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP%yšmœbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP:è.) _wandbruntime‚43  _runtime‚43™×#ANž€O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7397875425280 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1847985796608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385462 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7397892464640 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7412365087078 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 51547677030 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 51627787264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702768640 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1848078485504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055055 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚96795136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 QGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93128 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚109156 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚461154 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚451528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966447 trainer/global_step‚49 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7397892464640 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚108898 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1795416803328 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 52661682176 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 52587605504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843575517186508e+09 _step‚1 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 51286627840 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚451528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0T~"Š'Ÿ„O VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚109156 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚461154 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚451528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966447 trainer/global_step‚49 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7397892464640 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚108898 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1795416803328 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 52661682176 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 52587605504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843575517186508e+09 _step‚1 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 51286627840 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚451528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7397875425280 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1847985796608 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385462 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7397892464640 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7412365087078 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 51547677030 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 51627787264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702768640 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1848078485504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055055 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚96795136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 QGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93128çUím bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP¾cb¡b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP» ì’i¢bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPžcýÔg£bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP¶fÚ,`¤b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPDü)Ñe¥b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP9 †ác¦b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPcˆ›Lm§bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP„$™u¨bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP%á )r©bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPâ.kòfªba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPV!Ä4e«b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP6ôn¬bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPÁÉ·ˆm­bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP¾þæ]®bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP!˜€Þi¯bd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP¶ä&q°bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPw¨¿j±be BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPëÔf²ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPÊz·]³bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPY£Go´bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPóK kµbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÂëHd¶b_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPÏ"Šh¿bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP~%ä^ÀbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP¬ìÀ¡wÁbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPaU+ jÂbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPqàDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP¹‰‚}jÄbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP]JTggÅbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPÜçdfÆba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPû~RbÇb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPf3gaÈb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPž¥ýÙmÉbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPbœ˜kÊbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP²XHd^ËbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPÊj›àqÌbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPéÝHkÍbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPàÓ÷olÎbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPÚ»°wÏbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP-êßxoÐbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPïmâ(cÑb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPH¨ÝhÒbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPýúgŒjÓbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPiY"…jÔbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP‰ùb$lÕbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP ˜¼¾hÖbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPO@h’o×bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP¯‰B@mØbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP¬d_ÙbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP'ÓÚeÚb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPwØ•qÛbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP Ð_ÜbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPHÜþØrÝbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPðš¾jÞbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP‘ dßb_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPBÝßÆoäbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPòüÛ˜kåbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPƒ—J¬læbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP†âiçbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP ûÒ6rèbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPš—p%rébm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPè&úlêbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP Ûàqëbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP×SÕýmìbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPùÔùníbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÛ/så`îb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPnö}dïb_ köbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPò@¡f÷ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP+·¶iøbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPõ sx^ùbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPVjÑoúbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPØC£;oûbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPÿEIò\übW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP7O¯Épýbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP„ðòõeþb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP#…’gÿbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPÃz+·m€bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP‡ËÜibd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPT„²l‚bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPB¸Ümƒbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPkCäj„be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPÎWQoh…bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JP89Œf†ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP¯;è%b‡b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP@ÁÍßnˆbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP¬7³Îg‰bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP@Î÷ÙhŠbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP4FEn‹bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPñ1uŒbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPäŽa“rbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP=ʤsŽbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPù¨ibd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JP8OF$gbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP\Ì"Oe‘b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPð÷ƒœm’bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPïejUd“b_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚49 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1828845561856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚470220 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1882441566208 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 53697074688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚206065 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 SIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94876 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 52640421376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7559886812784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393094 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7545133494272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7545116454912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceSÍ›± tatsMonitor.on_train_batch_end/inactive_split.all.peak‚513 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7597415709696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚393092 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7597773915648 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 52218182390 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7507311042560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 53614616576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169T;T•&M ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 52640421376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7559886812784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393094 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7545133494272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7545116454912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚853672 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚513 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7597415709696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚393092 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7597773915648 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 52218182390 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7507311042560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 53614616576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 52299254784 OEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94873 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 % _timestamp‚1.7843575523342612e+09 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚111189 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚49 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1828845561856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚470220 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1882441566208 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 53697074688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚206065 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 SIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94876 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ÇVJAžb< loss/FRt1/assignment_loss"trainer/global_step2:JPl$&‚CŸb> loss/SRqqt2/assignment_loss"trainer/global_step2:JP´',A b< loss/FBt2/assignment_loss"trainer/global_step2:JP°”qb@¡b; loss/FBt2/detection_loss"trainer/global_step2:JPzrâ7¢b2 loss/total_loss"trainer/global_step2:JPó@ãÖ@£b; loss/FRt2/detection_loss"trainer/global_step2:JP¶D=WA¤b< loss/FRt2/assignment_loss"trainer/global_step2:JPO ¤@¥b; loss/FRt1/detection_loss"trainer/global_step2:JPtÊäïB¦b= loss/SRqqt1/detection_loss"trainer/global_step2:JPŸ&9C§b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPÈr­ÙA¨b< loss/FBt1/assignment_loss"trainer/global_step2:JP†|MB©b= loss/SRqqt2/detection_loss"trainer/global_step2:JPƒ51@ªb; loss/FBt1/detection_loss"trainer/global_step2:JP9½_u-«b( epoch"trainer/global_step2:JPø ä.¬) _wandbruntime‚44  _runtime‚44.CJø­ò /loss/FRt1/assignment_loss‚4.515418221642663 /loss/SRqqt2/assignment_loss‚4.2037642530992 0loss/FBt2/assignment_loss‚1.8802955001592636 /loss/FBt2/detection_loss‚35.568934578821064 %loss/total_loss‚78.28387813679647 trainer/global_step‚49 /loss/FRt2/detection_loss‚1.4859245031525754 _step‚3 /loss/FRt2/assignment_loss‚5.325399735181883 0loss/FRt1/detection_loss‚0.40129412276530146 0loss/SRqqt1/detection_loss‚7.350048779952721 % _timestamp‚1.7843575523379664e+09 0loss/SRqqt1/assignment_loss‚4.22687343300366 0loss/FBt1/assignment_loss‚1.8477785719765558 1loss/SRqqt2/detection_loss‚1.2247270048364227 /loss/FBt1/detection_loss‚10.253419432205163 epoch‚0 _runtime‚ 44.413020918¤À_Éü®ö _runtime‚ 44.413020918 /loss/FRt1/assignment_loss‚4.515418221642663 /loss/SRqqt2/assignment_loss‚4.2037642530992 0loss/FBt2/assignment_loss‚1.8802955001592636 /loss/FBt2/detection_loss‚35.568934578821064 %loss/total_loss‚78.28387813679647 trainer/global_step‚49 /loss/FRt2/detection_loss‚1.4859245031525754 _step‚3 /loss/FRt2/assignment_loss‚5.325399735181883 0loss/FRt1/detection_loss‚0.40129412276530146 0loss/SRqqt1/detection_loss‚7.350048779952721 % _timestamp‚1.7843575523379664e+09 0loss/SRqqt1/assignment_loss‚4.22687343300366 0loss/FBt1/assignment_loss‚1.8477785719765558 1loss/SRqqt2/detection_loss‚1.2247270048364227 /loss/FBt1/detection_loss‚10.253419432205163 epoch‚0‹¼.ا¯‚(:œ °ÅìÒ¤±€½ disk.sda.in‚0 disk.sda3.out‚ 1.19140625 disk.drbd2165.in‚0 disk.dm-0.in‚ 40.60546875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2638.75390625 network.sent‚46741 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 10.603515625$ disk./.usageGB‚585.6281280517578 disk.rbd0.out‚0 disk.sda.out‚ 1.19140625 disk.sda2.in‚0 disk.drbd2165.out‚0 network.recv‚49481 disk.rbd1.in‚0 disk.rbd1.out‚0.0625! disk.nvme0n1.out‚ 10.603515625( proc.memory.availableMB‚ 602.24609375 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚96.32418155670166 cpu‚26.181227217446814! disk.nvme0c0n1.in‚ 40.60546875# disk.nvme0c0n1.out‚ 10.603515625 disk.nvme0n1.in‚ 40.60546875) proc.memory.percent‚16.10567569732666 proc.cpu.threads‚17) disk./.usagePercent‚39.30199504828179 disk.dm-1.out‚ 1.19140625ÒÝàŸ°‚(:” °ÅìÒ•èì¼ gpu.0.gpu‚88.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ãu”±‚(:‰ ¿ÅìÒ×Û‡» gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Ùb‰®²‚(:£ ¿ÅìÒͤã¼ disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.30199504828179 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 157.06640625$ memory_percent‚96.32401466369629# proc.memory.rssMB‚ 2638.76171875 disk.sda2.in‚0 cpu‚26.16099404937629 network.sent‚100202 network.recv‚56793) proc.memory.percent‚16.10572338104248 disk.dm-0.out‚15.03466796875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 157.06640625 disk.sda.in‚0 disk.sda.out‚ 1.44921875 disk.rbd0.in‚0 disk.dm-0.in‚ 157.06640625 disk.rbd1.in‚0# disk.nvme0n1.out‚15.03466796875 disk.sda3.out‚ 1.44921875' proc.memory.availableMB‚ 602.2734375$ disk./.usageGB‚585.6281280517578 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 1.44921875 disk.rbd1.out‚0.125% disk.nvme0c0n1.out‚15.03466796875 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚00¤íž³‚(:“ ÎÅìÒÉé–» gpu.0.gpu‚41.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"tŠ¢´‚(:— ÎÅìÒ˺Ҿ disk.sda1.out‚0 disk.sda2.in‚0 cpu‚26.150757733381976 disk.nvme0n1.out‚ 17.8515625( proc.memory.availableMB‚ 601.73828125 disk.rbd1.out‚0.125 disk.dm-0.out‚ 17.8515625! disk.nvme0c0n1.out‚ 17.8515625" proc.memory.rssMB‚ 2638.7734375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 157.06640625 network.recv‚60771( disk./.usagePercent‚39.3019883920859 disk.drbd2165.in‚0 disk.dm-0.in‚ 157.06640625 disk.sda3.out‚ 1.69140625 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0$ memory_percent‚96.32728099822998 proc.cpu.threads‚17 disk.sda.in‚0 network.sent‚105784 disk.dm-1.out‚ 1.69140625 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.10579490661621$ disk./.usageGB‚585.6280288696289 disk.drbd2165.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 157.06640625 disk.sda.out‚ 1.691406256¥$Ê.µ) _wandbruntime‚84  _runtime‚84¡ô$i¶¼ ' lr-AdamW/pg1‚0.00010766244456913052 ' lr-AdamW/pg2‚0.00010766244456913052 trainer/global_step‚99 $ _timestamp‚1.784357592679947e+09 _runtime‚ 84.758220904 _step‚4îè}Æ·À $ _timestamp‚1.784357592679947e+09 _runtime‚ 84.758220904 _step‚4 ' lr-AdamW/pg1‚0.00010766244456913052 ' lr-AdamW/pg2‚0.00010766244456913052 trainer/global_step‚99.wŸ–.¸) _wandbruntime‚84  _runtime‚84ìo‰k¢'¹œO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14862267235328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14686333964288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 % _timestamp‚1.7843575926837611e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚915606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚915606 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚767062 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3466836117504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚394378 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚394121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 102164163938 trainer/global_step‚99 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 104025531392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14759926906880 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14759926906880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1673040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚905980 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14861909036544 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 101982129664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 104114288128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚394378 K@DeviceStatsMonitor.on_train_ P¨¸§batch_start/inactive_split.all.freed‚394121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 102164163938 trainer/global_step‚99 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 104025531392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14759926906880 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14759926906880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1673040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚905980 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14861909036544 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 101982129664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 104114288128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14686333964288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 % _timestamp‚1.7843575926837611e+09Ì1.») _wandbruntime‚85  _runtime‚852Äk=¬&¼¦M I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774692 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 103335923200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86652416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14936036893292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14907184975872 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14907167936512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 _runtime‚ 85.373782477 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774692 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3605318577664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102834669298 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3500283488256 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚915032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _step‚6 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚924672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚182123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 103175217772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15010162693120 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14832861675520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15010520899072 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚398211 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚182126 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1699366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14907167936512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÊE[ˆ°&½ªM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14832861675520 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15010520899072 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚398211 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚182126 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1699366 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14907167936512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774692 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 103335923200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86652416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14936036893292 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14907184975872 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14907167936512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 _runtime‚ 85.373782477 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774692 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3605318577664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102834669298 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3500283488256 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚915032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _step‚6 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1689724 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚924672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚182123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 103175217772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15010162693120s \q.¾) _wandbruntime‚85  _runtime‚85³4«Uù¿ó .loss/FBt1/detection_loss‚7.912449964288638 trainer/global_step‚99 % _timestamp‚1.7843575933009508e+09 _step‚7 -loss/FRt1/assignment_loss‚4.4982978482958 2loss/SRqqt1/assignment_loss‚2.9627656079410167 /loss/FRt2/detection_loss‚1.4814750565489134 1loss/SRqqt2/detection_loss‚1.1125295205775947 %loss/total_loss‚68.00027952277641 epoch‚0 /loss/FRt2/assignment_loss‚5.354794429779052 1loss/FBt1/assignment_loss‚0.22638028350368303 0loss/FBt2/assignment_loss‚0.4190659569803534 _runtime‚ 85.376045329 /loss/FRt1/detection_loss‚0.3183437785158236 .loss/FBt2/detection_loss‚33.60967988634437 2loss/SRqqt2/assignment_loss‚2.8610420575107103 0loss/SRqqt1/detection_loss‚7.243455132490464÷œ9ýÀ÷ .loss/FBt1/detection_loss‚7.912449964288638 trainer/global_step‚99 % _timestamp‚1.7843575933009508e+09 _step‚7 -loss/FRt1/assignment_loss‚4.4982978482958 2loss/SRqqt1/assignment_loss‚2.9627656079410167 /loss/FRt2/detection_loss‚1.4814750565489134 1loss/SRqqt2/detection_loss‚1.1125295205775947 %loss/total_loss‚68.00027952277641 epoch‚0 /loss/FRt2/assignment_loss‚5.354794429779052 1loss/FBt1/assignment_loss‚0.22638028350368303 0loss/FBt2/assignment_loss‚0.4190659569803534 _runtime‚ 85.376045329 /loss/FRt1/detection_loss‚0.3183437785158236 .loss/FBt2/detection_loss‚33.60967988634437 2loss/SRqqt2/assignment_loss‚2.8610420575107103 0loss/SRqqt1/detection_loss‚7.243455132490464ð.EÛŸÁ‚(:” ÝÅìÒù¤‘» gpu.0.gpu‚88.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.821799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P뺥‚(:š ÝÅìÒžÎê¼% proc.memory.availableMB‚ 601.40625 proc.memory.rssMB‚ 2638.78125 disk.rbd1.out‚0.1875) disk./.usagePercent‚39.30200452056056 disk.rbd0.in‚0% disk.nvme0c0n1.out‚22.35107421875 disk.nvme0n1.in‚ 157.06640625 disk.sda2.in‚0 network.sent‚111220 disk.sda3.in‚0$ disk./.usageGB‚585.6282691955566 disk.sda1.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17# disk.nvme0n1.out‚22.35107421875 disk.sda.out‚1.921875$ memory_percent‚96.32930755615234 network.recv‚64734 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚26.127472638463363" disk.nvme0c0n1.in‚ 157.06640625 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.out‚22.35107421875) proc.memory.percent‚16.10584259033203 disk.sda3.out‚1.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 157.06640625 disk.dm-1.in‚0 disk.dm-1.out‚1.921875Ä©«ŸÃ‚(:” ìÅìÒ®­» gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚148.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.662399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¦&(©Ä‚(:ž ìÅìÒø±÷¼ disk.drbd2165.in‚0 disk.dm-0.in‚ 157.06640625! disk.nvme0n1.out‚ 25.447265625 disk.sda.in‚0$ memory_percent‚96.32983207702637$ disk./.usageGB‚585.6281852722168 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.out‚ 2.56640625 disk.rbd1.out‚0.1875 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 157.06640625' proc.memory.availableMB‚ 601.3203125" proc.memory.rssMB‚ 2638.7890625) proc.memory.percent‚16.10589027404785 network.recv‚69679 disk.dm-1.in‚0 cpu‚26.10878347339481 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 157.06640625 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚39.301998888394806 disk.dm-0.out‚ 25.447265625 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 25.447265625 disk.sda.out‚ 2.56640625 disk.sda3.out‚ 2.56640625 network.sent‚158760ebdn’Å‚(:‡ ûÅìÒÝÏ» gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊN#ä2Æ‚(:’ ûÅìÒ¡ºè¼ disk.drbd2165.in‚0 disk.dm-0.out‚ 30.5478515625) proc.memory.percent‚16.10591411590576 network.sent‚164195 disk.sda.out‚2.765625( proc.memory.availableMB‚ 601.27734375 disk.dm-1.in‚0" disk.nvme0n1.out‚ 30.5478515625 disk.sda.in‚0 disk.sdaÀsCk2.out‚0 disk.dm-0.in‚ 261.59375# proc.memory.rssMB‚ 2638.79296875 disk.rbd1.in‚0 disk.rbd1.out‚0.25 disk.rbd0.in‚0 disk.nvme0n1.in‚ 261.59375 disk.sda3.out‚2.765625 disk.sda1.out‚0$ memory_percent‚96.33009433746338 cpu‚26.097523831812392 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 261.59375 disk.sda2.in‚0 proc.cpu.threads‚17# disk./.usageGB‚585.628246307373$ disk.nvme0c0n1.out‚ 30.5478515625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚2.765625 disk.sda3.in‚0 network.recv‚73789* disk./.usagePercent‚39.302002984515354FÐK0Ç+ _wandbruntime‚125  _runtime‚125U•ÛÅÈ¿ trainer/global_step‚149 % _timestamp‚1.7843576335737283e+09 _runtime‚ 125.649194365 _step‚8 ' lr-AdamW/pg1‚0.00016203741657374189 ' lr-AdamW/pg2‚0.00016203741657374189,ÑQÉÉà ' lr-AdamW/pg1‚0.00016203741657374189 ' lr-AdamW/pg2‚0.00016203741657374189 trainer/global_step‚149 % _timestamp‚1.7843576335737283e+09 _runtime‚ 125.649194365 _step‚8\§ß¡0Ê+ _wandbruntime‚125  _runtime‚125ƒN«'Ë¥O _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚149 % _timestamp‚1.7843576335747454e+09 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1370058 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 152677631488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22274656019968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 155550765056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1148660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1370058 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22164307660372 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22011867557888ýí4¯'Ì©O _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5138274043904 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 155459911168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22121978388480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2509092 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1360432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 152780650846 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚149 % _timestamp‚1.7843576335747454e+09 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1370058 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 152677631488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22274656019968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 155550765056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1148660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1370058 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22164307660372 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22011867557888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 HDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1156292 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22269219418112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 153451156206 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚321488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22158378229760 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1156294 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5171702802432 _step‚10 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 trainer/global_step‚149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1369484 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154031425024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22423267882496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22311829385966 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5171721414656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚269373 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5328291491328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 153690258432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚321215 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 154031425024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22269236457472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 % _timestamp‚1.7843576341880994e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚269376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273¤€z¯ ϵM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22311829385966 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5171721414656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚269373 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5328291491328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCÏ ° ›!DeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 153690258432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚321215 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 154031425024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22269236457472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 % _timestamp‚1.7843576341880994e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚269376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1379124 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 156570076672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1156292 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 153791704680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1156292 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22269219418112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 153451156206 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚321488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22158378229760 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1156294 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5171702802432 _step‚10 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 trainer/global_step‚149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1369484 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154031425024 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22423267882496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 :˜}•0Ð+ _wandbruntime‚126  _runtime‚126}ø*úÑô 0loss/SRqqt1/detection_loss‚5.755611763305442 /loss/FBt2/detection_loss‚14.306284799248159 epoch‚0 0loss/FRt1/detection_loss‚0.42450762580253615 .loss/FBt1/detection_loss‚5.241383235048004 % _timestamp‚1.7843576341904805e+09 1loss/SRqqt1/assignment_loss‚2.483789180600366 0loss/FBt1/assignment_loss‚0.2638874972783794 .loss/FRt2/detection_loss‚1.465382775979114 #loss/total_loss‚43.105919568584 _step‚11 .loss/FRt1/assignment_loss‚4.55161543274981 /loss/FRt2/assignment_loss‚5.332063571579336 1loss/FBt2/assignment_loss‚0.14485725251793424 1loss/SRqqt2/detection_loss‚0.8291509433848075 trainer/global_step‚149 _runtime‚ 126.265560193 1loss/SRqqt2/assignment_loss‚2.307385491090109ÎŒ)þÒø .loss/FRt1/assignment_loss‚4.55161543274981 /loss/FRt2/assignment_loss‚5.332063571579336 1loss/FBt2/assignment_loss‚0.14485725251793424 1loss/SRqqt2/detection_loss‚0.8291509433848075 trainer/global_step‚149 _runtime‚ 126.265560193 1loss/SRqqt2/assignment_loss‚2.307385491090109 0loss/SRqqt1/detection_loss‚5.755611763305442 /loss/FBt2/detection_loss‚14.306284799248159 epoch‚0 0loss/FRt1/detection_loss‚0.42450762580253615 .loss/FBt1/detection_loss‚5.241383235048004 % _timestamp‚1.7843576341904805e+09 1loss/SRqqt1/assignment_loss‚2.483789180600366 0loss/FBt1/assignment_loss‚0.2638874972783794 .loss/FRt2/detection_loss‚1.465382775979114 #loss/total_loss‚43.105919568584 _step‚11 —Èîì”Ó‚(:‰ ŠÆìÒ‘™Ã¼ gpu.0.gpu‚57.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜHdj›Ô‚(: ŠÆìÒö‚—½ disk.rbd1.in‚0$ disk./.usageGB‚585.6282920837402 disk.sda.in‚0 disk.dm-1.out‚ 3.08984375 disk.nvme0n1.in‚ 261.59375! disk.nvme0n1.out‚ 34.775390625$ memory_percent‚96.33502960205078 cpu‚26.09135660105376 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚39.30200605660576 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚0.25 disk.rbd0.in‚0* proc.memory.percent‚16.105937957763672 disk.sda1.in‚0 disk.dm-0.in‚ 261.59375 disk.nvme0c0n1.in‚ 261.59375# disk.nvme0c0n1.out‚ 34.775390625 proc.cpu.threads‚17 network.recv‚77924 disk.sda.out‚ 3.08984375 disk.sda2.in‚0 disk.dm-0.out‚ 34.775390625 network.sent‚169639 disk.sda3.out‚ 3.08984375 disk.rbd0.out‚0% proc.memory.availableMB‚ 600.46875! proc.memory.rssMB‚ 2638.796875›£×’Õ‚(:‡ ™ÆìÒþ÷Ѽ gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~½Î–žÖ‚(:“ ™ÆìÒ¾åü½) disk./.usagePercent‚39.30201066474138 disk.sda1.out‚0 proc.cpu.threads‚17# disk./.usageGB‚585.628360748291 disk.sda.out‚ 3.3046875 disk.sda3.out‚ 3.3046875 disk.dm-0.out‚ 43.3212890625 cpu‚26.081813433157144 disk.nvme0n1.in‚ 261.59375 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 261.59375$ disk.nvme0c0n1.out‚ 43.3212890625& proc.memory.availableMB‚ 600.546875 disk.dm-1.out‚ 3.3046875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.out‚ 43.3212890625 disk.rbd1.out‚0.375 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚96.33455276489258 disk.sda2.in‚0 network.sent‚217418* proc.memory.percent‚16.105985641479492 network.recv‚83017 disk.sda.in‚0 disk.dm-0.in‚ 261.59375" proc.memory.rssMB‚ 2638.8046875fÞ<òž×‚(:“ ¨ÆìÒö¢Ö» gpu.0.gpu‚69.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯hªŸØ‚(:” ¨ÆìÒ”™è¼ disk.dm-0.out‚ 47.6044921875 disk.rbd1.in‚0 disk.rbd1.out‚0.375$ disk./.usageGB‚585.6282424926758 disk.rbd0.in‚0 disk.sda3.out‚3.671875 disk.sda2.in‚0 disk.dm-0.in‚ 261.59375* proc.memory.percent‚16.105985641479492 disk.nvme0c0n1.in‚ 261.59375 disk.sda3.in‚0( proc.memory.availableMB‚ 602.82421875 cpu‚26.086230727640366$ disk.nvme0c0n1.out‚ 47.6044921875 disk.sda.out‚3.671875 proc.cpu.threads‚17 network.sent‚222020 disk.rbd0.out‚0" disk.nvme0n1.out‚ 47.6044921875 disk.dm-1.out‚3.671875$ memory_percent‚96.32065296173096* disk./.usagePercent‚39.302002728507816 disk.sda1.out‚0 disk.drbd2165.out‚0 network.recv‚86258 disk.nvme0n1.in‚ 261.59375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2638.8046875¨ê¯+0Ù+ _wandbruntime‚166  _runtime‚166®7dÆÚÀ ' lr-AdamW/pg1‚0.00021641238857835325 ' lr-AdamW/pg2‚0.00021641238857835325 trainer/global_step‚199 % _timestamp‚1.7843576743887787e+09 _runtime‚ 166.464222596 _step‚12 "N&ÊÛÄ trainer/global_step‚199 % _timestamp‚1.7843576743887787e+09 _runtime‚ 166.464222596 _step‚12 ' lr-AdamW/pg1‚0.00021641238857835325 ' lr-AdamW/pg2‚0.00021641238857835325 ÊC<0Ü+ _wandbruntime‚166  _runtime‚166¶Rœë­'ݧO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203056589392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29540457740880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 % _timestamp‚1.7843576743895833e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚423946 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29687761202176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29484029870080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7016635296256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29687403003392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7016527927296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 203397137754 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1814884 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1814884 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29337401151488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29484046909440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1824510 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1530260 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 203373133312 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203714292736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29687761202176 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1530260 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚423668 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6809730582528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚354881 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1824510 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6809711970304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 206815956992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚778827 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 166.465074665 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚778546 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 203714292736 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚354878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 trainer/global_step‚199 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1530262 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29484046909440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29484029870080 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1530262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088íPT¿”Þ«O \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7016635296256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29687403003392 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7016527927296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 203397137754 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1814884 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1814884 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29337401151488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29484046909440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1824510 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1530260 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 203373133312 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203714292736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29687761202176 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1530260 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚423668 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6809730582528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚354881 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1824510 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6809711970304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 206815956992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚778827 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚ 166.465074665 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚778546 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 203714292736 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚354878 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 trainer/global_step‚199 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1530262 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29484046909440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29484029870080 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1530262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 WJDeviceStatsMonitor.on_train_bjD  atch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203056589392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29540457740880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 % _timestamp‚1.7843576743895833e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚423946 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29687761202176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29484029870080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 yg-'0ß+ _wandbruntime‚167  _runtime‚167›$«ê¶&à°M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1537892 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚356623 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537894 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88749568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3361828 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204726926848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29483928862720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 207834112512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1833576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204408191588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1823936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29836014865920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29836014865920 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚426058 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29631270899712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29631270899712 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7051082203136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204726926848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207922862080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207922862080 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1537892 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚356623 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537894 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88749568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3361828 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204726926848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29483928862720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 207834112512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1833576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204408191588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1823936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29836014865920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29836014865920 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚426058 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29631270899712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29631270899712 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7051082203136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204726926848\ë…®0â+ _wandbruntime‚167  _runtime‚167&?¥þãø 1loss/SRqqt1/detection_loss‚4.9696268964886166 .loss/FBt1/detection_loss‚9.055215945185013 .loss/FBt2/detection_loss‚9.413334160989862 %loss/total_loss‚40.97980977694647 % _timestamp‚1.7843576750082176e+09 trainer/global_step‚199 _runtime‚ 167.083296151 epoch‚0 /loss/FRt1/assignment_loss‚4.619312264227098 2loss/SRqqt2/assignment_loss‚2.2280398888465687 1loss/SRqqt2/detection_loss‚0.9307174702619728 _step‚15 /loss/FRt2/assignment_loss‚4.655884306944773 /loss/FBt1/assignment_loss‚0.259289258917902 1loss/FBt2/assignment_loss‚0.19996876242019387 /loss/FRt1/detection_loss‚0.5367752748494029 /loss/FRt2/detection_loss‚1.6467792113955706 2loss/SRqqt1/assignment_loss‚2.4648663364194925Oh¹äü /loss/FRt2/assignment_loss‚4.655884306944773 /loss/FBt1/assignment_loss‚0.259289258917902 1loss/FBt2/assignment_loss‚0.19996876242019387 /loss/FRt1/detection_loss‚0.5367752748494029 /loss/FRt2/detection_loss‚1.6467792113955706 2loss/SRqqt1/assignment_loss‚2.4648663364194925 1loss/SRqqt1/detection_loss‚4.9696268964886166 .loss/FBt1/detection_loss‚9.055215945185013 .loss/FBt2/detection_loss‚9.413334160989862 %loss/total_loss‚40.97980977694647 % _timestamp‚1.7843576750082176e+09 trainer/global_step‚199 _runtime‚ 167.083296151 epoch‚0 /loss/FRt1/assignment_loss‚4.619312264227098 2loss/SRqqt2/assignment_loss‚2.2280398888465687 1loss/SRqqt2/detection_loss‚0.9307174702619728 _step‚15º©í€Ÿå‚(:” ·ÆìÒÛ°í¼ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0• JA«æ‚(:  ·ÆìÒ©Ç‹½! disk.nvme0c0n1.in‚ 376.6640625% disk.nvme0c0n1.out‚54.10107421875 disk.sda.in‚0 cpu‚26.07616490666097$ disk./.usageGB‚585.6282272338867 disk.sda1.in‚0 disk.rbd1.out‚0.4375 disk.sda3.in‚0# proc.memory.rssMB‚ 2638.80859375 network.sent‚227371 disk.dm-1.in‚0$ memory_percent‚96.32360935211182( proc.memory.availableMB‚ 602.33984375 disk.dm-0.out‚54.10107421875 disk.dm-1.out‚ 3.9765625 disk.drbd2165.out‚0 disk.dm-0.in‚ 376.6640625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 376.6640625# disk.nvme0n1.out‚54.10107421875 disk.sda.out‚ 3.9765625 disk.sda2.out‚0 disk.sda3.out‚ 3.9765625 disk.sda1.out‚0 network.recv‚90650 disk.rbd0.out‚0* proc.memory.percent‚16.106009483337402 proc.cpu.threads‚17) disk./.usagePercent‚39.30200170447768 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0×7fç”ç‚(:‰ ÆÆìÒ—˜Çº gpu.0.gpu‚57.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[TZ‹­è‚(:¢ ÆÆìÒã麼 disk.rbd0.out‚0 disk.dm-0.out‚56.92041015625 network.sent‚274723 disk.dm-1.in‚0# proc.memory.rssMB‚ 2638.80859375$ disk./.usageGB‚585.6282501220703 disk.rbd1.in‚0 disk.nvme0n1.in‚ 376.6640625 disk.sda.out‚ 4.26953125 disk.sda3.in‚0 disk.sda3.out‚ 4.26953125* disk./.usagePercent‚39.302003240522886! disk.nvme0c0n1.in‚ 376.6640625 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 4.26953125 proc.cpu.threads‚17# disk.nvme0n1.out‚56.92041015625 cpu‚26.06768832417113 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 376.6640625# memory_percent‚96.3263988494873* proc.memory.percent‚16.106009483337402% disk.nvme0c0n1.out‚56.92041015625' proc.memory.availableMB‚ 601.8828125 network.recv‚95214 disk.rbd1.out‚0.4375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0°§C0é+ _wandbruntime‚207  _runtime‚207ĶÕLÆêÀ ' lr-AdamW/pg1‚0.00027078736058296467 ' lr-AdamW/pg2‚0.00027078736058296467 trainer/global_step‚249 % _timestamp‚1.7843577151688263e+09 _runtime‚ 207.244372761 _step‚16¸nÏ¿ÊëÄ ' lr-AdamW/pg1‚0.00027078736058296467 ' lr-AdamW/pg2‚0.00027078736058296467 trainer/global_step‚249 % _timestamp‚1.7843577151688263e+09 _runtime‚ 207.244372761 _step‚16‡éT0ì+ _wandbruntime‚207  _runtime‚207½Ò „(í¦O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257974641664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1911862 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1911860 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚527916 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95048192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8481149896704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36846081351680 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 254409794560 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚442131 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 254068635136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36846098391040 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1911862 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚970044 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37100508185600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257879593472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36662934745088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 trainer/global_step‚249 $ _timestamp‚1.784357715172534e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1911860 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36916965409110 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36662951784448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113660416 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8739029490176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚970292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚ 207.248264045 TDDeviceStatsMonitor.o§wR„n_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528161 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 254013624662 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36662951784448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113660416 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8739029490176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚970292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚ 207.248264045 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528161 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 254013624662 HDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 255422428672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8514597267456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36809445416960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 254684130022 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36993339420672 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1919494 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36993322381312 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 255081262080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 % _timestamp‚1.7843577157864547e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37064129546982 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36809462456320 _step‚18 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95041024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37064487134816 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2288028 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113653248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36993339420672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1919492 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚443873 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37248403643392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚974066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2278388 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 255422428672 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736:ú,íÂñµM TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8514597267456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36809445416960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 254684130022 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36993339420672 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1919494 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36993322381312 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 255081262080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 % _timestamp‚1.7843577157864547e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37064129546982 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36809462456320 _step‚18 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95041024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37064487134816 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2288028 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113653248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36993339420672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1919492 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚443873 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37248403643392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚974066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2278388 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 255422428672 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 255422428672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 D$0ò+ _wandbruntime‚207  _runtime‚207ÆךDüóö % _timestamp‚1.7843577157896225e+09 _runtime‚ 207.864706072 /loss/FRt1/assignment_loss‚4.564890547127559 -loss/FRt2/detection_loss‚1.46003434793591 _step‚19 1loss/SRqqt1/detection_loss‚3.2889185642654244 %loss/total_loss‚35.81693257974385 epoch‚0 trainer/global_step‚249 0loss/FRt2/assignment_loss‚4.6139577486253085 2loss/SRqqt1/assignment_loss‚1.4773911291902715 0loss/FBt1/assignment_loss‚0.2650919735007454 0loss/FBt2/assignment_loss‚0.2538972696189583 0loss/FRt1/detection_loss‚0.36337913898887625 /loss/FBt1/detection_loss‚10.087725868144396 1loss/SRqqt2/assignment_loss‚2.190847976099361 1loss/SRqqt2/detection_loss‚1.2006282878033359 ,loss/FBt2/detection_loss‚6.0501697284437 è¬ôú /loss/FRt1/assignment_loss‚4.564890547127559 -loss/FRt2/detection_loss‚1.46003434793591 _step‚19 1loss/SRqqt1/detection_loss‚3.2889185642654244 %loss/total_loss‚35.81693257974385 epoch‚0 trainer/global_step‚249 0loss/FRt2/assignment_loss‚4.6139577486253085 2loss/SRqqt1/assignment_loss‚1.4773911291902715 0loss/FBt1/assignment_loss‚0.2650919735007454 0loss/FBt2/assignment_loss‚0.2538972696189583 0loss/FRt1/detection_loss‚0.36337913898887625 /loss/FBt1/detection_loss‚10.087725868144396 1loss/SRqqt2/assignment_loss‚2.190847976099361 1loss/SRqqt2/detection_loss‚1.2006282878033359 ,loss/FBt2/detection_loss‚6.0501697284437 % _timestamp‚1.7843577157896225e+09 _runtime‚ 207.864706072†; LŸõ‚(:” ÕÆìÒÉÚ¼ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ápœö‚(:‘ ÕÆìÒÌ솾 network.sent‚280169 disk.sda.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 4.6328125 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 376.6640625 disk.dm-0.out‚ 61.77734375( proc.memory.availableMB‚ 601.47265625 proc.memory.rssMB‚ 2638.8125 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 61.77734375 disk.nvme0n1.in‚ 376.6640625 disk.sda3.out‚ 4.6328125 disk.dm-1.out‚ 4.6328125$ memory_percent‚96.32890224456787* proc.memory.percent‚16.106033325195312 network.recv‚99419) disk./.usagePercent‚39.30200759265097$ disk./.usageGB‚585.6283149719238 disk.drbd2165.in‚0 disk.rbd1.out‚0.5 disk.rbd0.out‚0 disk.nvme0n1.out‚ 61.77734375 disk.sda3.in‚0 disk.dm-1.in‚0 cpu‚26.059785907072612 disk.sda2.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 376.6640625èÊtž÷‚(:“ äÆìÒ”¦ã» gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q¸§º›ø‚(: äÆìÒØŽá¼ disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚585.6281929016113! disk.nvme0c0n1.in‚ 376.6640625 disk.sda1.in‚0 disk.dm-0.in‚ 376.6640625 proc.memory.rssMB‚ 2638.8125' proc.memory.availableMB‚ 601.5390625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚0.5! disk.nvme0c0n1.out‚ 65.6484375 disk.sda3.in‚0 disk.sda2.in‚0* proc.memory.percent‚16.106033325195312 cpu‚26.06048101872317 network.recv‚104259* disk./.usagePercent‚39.301999400409876 disk.nvme0n1.in‚ 376.6640625 disk.nvme0n1.out‚ 65.6484375 disk.dm-1.out‚ 4.98828125 network.sent‚327825 disk.rbd1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-0.out‚ 65.6484375# memory_percent‚96.3284969329834 disk.sda.out‚ 4.98828125 disk.sda3.out‚ 4.98828125IØœ;”ù‚(:‰ óÆìÒ¨˜¸» gpu.0.gpu‚82.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$6Ù«ú‚(:  óÆìÒÍ™˜½ disk.dm-0.out‚72.32470703125 disk.rbd1.out‚0.5625# disk.nvme0n1.out‚72.32470703125( proc.memory.availableMB‚ 601.54296875$ disk./.usageGB‚585.6283531188965 disk.drbd2165.out‚0 disk.dm-0.in‚ 499.9140625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 499.9140625 disk.sda.in‚0 network.sent‚333190 disk.sda3.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 499.9140625 network.recv‚108353 disk.sda3.out‚ 5.28515625 disk.sda1.out‚0$ memory_percent‚96.32847309112549 proc.cpu.threads‚17) disk./.usagePercent‚39.30201015272631 disk.sda1.in‚0 proc.memory.rssMB‚ 2638.8125 cpu‚26.05885137296994 disk.dm-1.out‚ 5.28515625% disk.nvme0c0n1.out‚72.32470703125 disk.sda.out‚ 5.28515625 disk.dm-1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.106033325195312 disk.sda2.out‚0m½sI0û+ _wandbruntime‚248  _runtime‚248½<ÞlÂü¼ % _timestamp‚1.7843577559397824e+09 _runtime‚ 248.015216984 _step‚20 % lr-AdamW/pg1‚0.000325162332587576 % lr-AdamW/pg2‚0.000325162332587576 trainer/global_step‚299ún‚ÆýÀ % lr-AdamW/pg2‚0.000325162332587576 trainer/global_step‚299 % _timestamp‚1.7843577559397824e+09 _runtime‚ 248.015216984 _step‚20 % lr-AdamW/pg1‚0.000325162332587576ÝŸ^0þ+ _wandbruntime‚248  _runtime‚248cýÓó'ÿ­O G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44512896970240 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305105296384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44293115489618 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10461746360832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44292757901896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44208132833280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44513255169024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2293460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2723788 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10152587823104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚632525 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10152606435328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43988485378048 % _timestamp‚1.7843577559408004e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2293460 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2293462 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2733414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚529381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44208132833280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2723788 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1161638 trainer/global_step‚299 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 309139925504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 304764136960 _runtime‚ 248.016344603 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2733414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2293462 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2™Xš›·'€±O G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2293460 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2293462 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2733414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚529381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44208132833280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2723788 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1161638 trainer/global_step‚299 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 309139925504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 304764136960 _runtime‚ 248.016344603 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2733414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2293462 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44512896970240 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305105296384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44293115489618 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10461746360832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44292757901896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44208132833280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44513255169024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2293460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2723788 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10152587823104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚632525 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10152606435328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43988485378048 % _timestamp‚1.7843577559408004e+09 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2301092 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88749568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 306117930496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44134996049920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2301094 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10496087334912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44440279627490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1165695 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚634290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44134979010560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44355373862912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2732840 _runtime‚ 248.631098898 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 310159502848 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44661150626816 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚634569 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚22 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10186035193856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305300616930 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚531126 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44355390902272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2742480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2732840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚22 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10186035193856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305300616930 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚531126 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44355390902272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2742480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2732840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2301092 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88749568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 306117930496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44134996049920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2301094 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10496087334912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44440279627490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1165695 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚634290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44134979010560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44355373862912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2732840 _runtime‚ 248.631098898Evé0„+ _wandbruntime‚248  _runtime‚248Kg«ý…÷ _runtime‚ 248.633163816 _step‚23 /loss/FRt2/assignment_loss‚4.488792543233945 1loss/SRqqt1/assignment_loss‚1.834974124110662 0loss/SRqqt2/detection_loss‚1.226137404807445 2loss/SRqqt2/assignment_loss‚1.6489887301811434 0loss/FBt1/assignment_loss‚0.4328811518619505 /loss/FRt2/detection_loss‚1.3733498253547283 1loss/SRqqt1/detection_loss‚3.1102372023276983 /loss/FRt1/assignment_loss‚4.482222892087082 /loss/FRt1/detection_loss‚0.4308861453163518 .loss/FBt1/detection_loss‚8.432255810864692 .loss/FBt2/detection_loss‚5.316647625831028 %loss/total_loss‚32.90832268272009 epoch‚0 trainer/global_step‚299 % _timestamp‚1.7843577565580711e+09 1loss/FBt2/assignment_loss‚0.13094922674336748sð¿ú†û /loss/FRt1/assignment_loss‚4.482222892087082 /loss/FRt1/detection_loss‚0.4308861453163518 .loss/FBt1/detection_loss‚8.432255810864692 .loss/FBt2/detection_loss‚5.316647625831028 %loss/total_loss‚32.90832268272009 epoch‚0 trainer/global_step‚299 % _timestamp‚1.7843577565580711e+09 1loss/FBt2/assignment_loss‚0.13094922674336748 _runtime‚ 248.633163816 _step‚23 /loss/FRt2/assignment_loss‚4.488792543233945 1loss/SRqqt1/assignment_loss‚1.834974124110662 0loss/SRqqt2/detection_loss‚1.226137404807445 2loss/SRqqt2/assignment_loss‚1.6489887301811434 0loss/FBt1/assignment_loss‚0.4328811518619505 /loss/FRt2/detection_loss‚1.3733498253547283 1loss/SRqqt1/detection_loss‚3.1102372023276983b÷bvŸ‡‚(:” ‚ÇìÒ‰¤» gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚259.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.821799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òé`«œˆ‚(:‘ ‚ÇìÒ‹Ò°¼ disk.drbd2165.in‚0 network.sent‚338892 disk.nvme0n1.in‚ 499.9140625 disk.sda3.in‚0 disk.drbd2165.out‚0 network.recv‚112573 disk.dm-1.out‚ 5.6640625! disk.nvme0c0n1.in‚ 499.9140625 disk.nvme0n1.out‚77.53125 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.106057167053223 disk.nvme0c0n1.out‚77.53125 disk.dm-0.out‚77.53125 disk.sda.in‚0 disk.sda.out‚ 5.6640625 disk.sda1.out‚0 cpu‚26.055849564356077$ disk./.usageGB‚585.6278266906738 disk.dm-0.in‚ 499.9140625 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.5625 disk.sda2.in‚0# proc.memory.rssMB‚ 2638.81640625$ memory_percent‚96.33047580718994* disk./.usagePercent‚39.301974823686585 disk.sda3.out‚ 5.6640625 disk.rbd0.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 601.21484375XÍÿ”‰‚(:‰ ‘ÇìÒƒ©§» gpu.0.gpu‚39.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å~@È­Š‚(:¢ ‘ÇìÒà£é» disk.dm-1.out‚ 6.00390625 proc.cpu.threads‚17 disk.rbd1.out‚0.625! disk.nvme0c0n1.in‚ 499.9140625# disk.nvme0n1.out‚84.42919921875 cpu‚26.05650488913329 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 499.9140625$ memory_percent‚96.33033275604248( proc.memory.availableMB‚ 601.23828125" proc.memory.rssMB‚ 2638.8203125 network.sent‚386275 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda.out‚ 6.00390625$ disk./.usageGB‚585.6288719177246 disk.nvme0n1.in‚ 499.9140625 disk.sda3.out‚ 6.00390625 disk.dm-0.out‚84.42919921875* proc.memory.percent‚16.106081008911133) disk./.usagePercent‚39.30204496975097% disk.nvme0c0n1.out‚84.42919921875 disk.dm-1.in‚0 network.recv‚117609 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0x’ÀF¤‹‚(:™  ÇìÒÓøº¼ network.sent‚391574* disk./.usagePercent‚39.302035241464665 disk.rbd0.in‚0! disk.nvme0n1.out‚ 87.623046875 disk.sda.out‚ 6.3046875 cpu‚26.056884467524995 disk.dm-1.out‚ 6.3046875! disk.nvme0c0n1.in‚ 499.9140625# disk.nvme0c0n1.out‚ 87.623046875" proc.memory.rssMB‚ 2638.8203125* proc.memory.percent‚16.106081008911133 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚0.625 disk.rbd1.in‚0 disk.sda3.out‚ 6.3046875& proc.memory.availableMB‚ 601.234375$ disk./.usageGB‚585.6287269592285 disk.sda2.in‚0 disk.dm-0.out‚ 87.623046875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 499.9140625 disk.nvme0n1.in‚ 499.9140625 disk.sda3.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 network.recv‚121343 disk.rbd0.out‚0 disk.sda.in‚0 disk.drbd2165.in‚0$ memory_percent‚96.33035659790039ƒhŸŒ‚(:”  ÇìÒþ™“½ gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚214.413' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.882600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m6¦0+ _wandbruntime‚288  _runtime‚288Nڀν & lr-AdamW/pg1‚0.0003795373045921874 & lr-AdamW/pg2‚0.0003795373045921874 trainer/global_step‚349 $ _timestamp‚1.784357796721746e+09 _runtime‚ 288.797069272 _step‚244mصÇÁ _runtime‚ 288.797069272 _step‚24 & lr-AdamW/pg1‚0.0003795373045921874 & lr-AdamW/pg2‚0.0003795373045921874 trainer/global_step‚349 $ _timestamp‚1.784357796721746e+09bèEÉ0+ _wandbruntime‚288  _runtime‚288vû̲'‘¬O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784357796722761e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2675062 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51925643953664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚25 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚616628 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1353705 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51926002152448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 355800798208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 355459638784 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2675060 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3187866 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3178240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 360356646400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51314018971648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51669265570126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51570201354240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚01"ü~¬’°O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51926002152448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 355800798208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 355459638784 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2675060 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3187866 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3178240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 360356646400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51314018971648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51669265570126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51570201354240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784357796722761e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2675062 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51925643953664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚25 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚616628 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1353705E+Ò0“+ _wandbruntime‚289  _runtime‚289Õ,<­½&”·M I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3187292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51816429707998 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 356257652312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 361375998976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚618373 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52074255816192 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3196932 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51717425344512 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚618376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51460512604160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51816787295832 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 % _timestamp‚1.7843577973345175e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52074255816192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12218733368832 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2682694 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 361278860800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚739446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51460529643520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2682694 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51717425344512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5869984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚739171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52073897610240 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1357822 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 356813432320 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 356472265728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115750400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11857473120256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2682692 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 355917103838 _step‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12218849119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 355917103838 _step‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12218849119232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3196932 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51717425344512 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚618376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51460512604160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51816787295832 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 % _timestamp‚1.7843577973345175e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52074255816192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12218733368832 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2682694 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 361278860800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚739446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51460529643520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2682694 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51717425344512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5869984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚739171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52073897610240 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1357822 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 356813432320 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 356472265728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115750400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11857473120256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0³eÇÅ0–+ _wandbruntime‚289  _runtime‚289°Çû—ü 0loss/FRt2/assignment_loss‚3.6293441363789967 0loss/FBt1/assignment_loss‚0.6283900887956695 .loss/FBt2/detection_loss‚5.834586361961923 % _timestamp‚1.7843577973369613e+09 0loss/FRt1/detection_loss‚0.42342404423623287 /loss/FRt2/detection_loss‚1.4219477188836518 1loss/SRqqt1/detection_loss‚2.9936014610022736 _step‚27 2loss/SRqqt1/assignment_loss‚1.9385259707502072 2loss/SRqqt2/assignment_loss‚1.5516228718874032 .loss/FBt1/detection_loss‚4.739698197163391 trainer/global_step‚349 /loss/FRt1/assignment_loss‚4.017756439693178 1loss/FBt2/assignment_loss‚0.12283544719891909 1loss/SRqqt2/detection_loss‚0.9763641912087674 &loss/total_loss‚28.278096929160615 epoch‚0 _runtime‚ 289.4120420054 18˜€ 0loss/FRt1/detection_loss‚0.42342404423623287 /loss/FRt2/detection_loss‚1.4219477188836518 1loss/SRqqt1/detection_loss‚2.9936014610022736 _step‚27 2loss/SRqqt1/assignment_loss‚1.9385259707502072 2loss/SRqqt2/assignment_loss‚1.5516228718874032 .loss/FBt1/detection_loss‚4.739698197163391 trainer/global_step‚349 /loss/FRt1/assignment_loss‚4.017756439693178 1loss/FBt2/assignment_loss‚0.12283544719891909 1loss/SRqqt2/detection_loss‚0.9763641912087674 &loss/total_loss‚28.278096929160615 epoch‚0 _runtime‚ 289.412042005 0loss/FRt2/assignment_loss‚3.6293441363789967 0loss/FBt1/assignment_loss‚0.6283900887956695 .loss/FBt2/detection_loss‚5.834586361961923 % _timestamp‚1.7843577973369613e+09úKVŽ”™‚(:‰ ¯ÇìÒñƒ¬» gpu.0.gpu‚88.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ër¸@œš‚(:‘ ¯ÇìÒø¬Ý¼ disk.nvme0n1.out‚ 93.78515625! proc.memory.rssMB‚ 2638.828125) disk./.usagePercent‚39.30204240967562 disk.sda.out‚ 6.70703125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 6.70703125 network.sent‚396812 disk.sda2.in‚0 disk.dm-0.out‚ 93.78515625 disk.dm-1.in‚0 network.recv‚125319" disk.nvme0c0n1.out‚ 93.78515625" memory_percent‚96.331787109375 disk.rbd1.in‚0# disk./.usageGB‚585.628833770752 disk.sda3.out‚ 6.70703125 proc.memory.availableMB‚601* proc.memory.percent‚16.106128692626953 disk.sda1.out‚0 disk.nvme0n1.in‚ 630.90234375 disk.rbd1.out‚0.6875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 630.90234375 cpu‚26.058832522756095 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 630.90234375EìÑ”›‚(:‰ ¾ÇìÒõ©±º gpu.0.gpu‚48.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥9ÿœ‚(:§ ¾Çìҙϼ disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 630.90234375% disk.nvme0c0n1.out‚96.77099609375 disk.sda.out‚ 7.07421875$ memory_percent‚96.33147716522217 network.sent‚444176 netàÛU%³work.recv‚129429 disk.drbd2165.in‚0* proc.memory.percent‚16.106176376342773 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 7.07421875( proc.memory.availableMB‚ 601.05078125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.out‚96.77099609375 disk.rbd1.out‚0.6875 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 7.07421875 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 630.90234375 disk.nvme0n1.in‚ 630.90234375# disk.nvme0n1.out‚96.77099609375" proc.memory.rssMB‚ 2638.8359375 cpu‚26.05698698877978* disk./.usagePercent‚39.301965863422886 disk.drbd2165.out‚0$ disk./.usageGB‚585.6276931762695«Ž„°0+ _wandbruntime‚329  _runtime‚329¿¡¿=Þ½ & lr-AdamW/pg1‚0.0004339122765967988 & lr-AdamW/pg2‚0.0004339122765967988 trainer/global_step‚399 % _timestamp‚1.7843578374817839e+09 _runtime‚ 329.56023634 _step‚28¢Ñ¥ÇŸÁ _runtime‚ 329.56023634 _step‚28 & lr-AdamW/pg1‚0.0004339122765967988 & lr-AdamW/pg2‚0.0004339122765967988 trainer/global_step‚399 % _timestamp‚1.7843578374817839e+09Ž,90 + _wandbruntime‚329  _runtime‚329­*lÁ³'¡­O bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 411515571712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59338749135872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59338390937088 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 _step‚29 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3632692 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1545670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59338749135872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13495482288128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3632692 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 405863085386 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚399 % _timestamp‚1.7843578374857712e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 406496300032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58932235796480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58639535525888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6698980 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚841789 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 406155140608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13906880005120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99242496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13906997859840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117854720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3056660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚703881 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13495463675904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3642318 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872šÍ„·'¢±O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59338749135872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13495482288128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3632692 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 405863085386 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚399 % _timestamp‚1.7843578374857712e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 406496300032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58932235796480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58639535525888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6698980 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚841789 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 406155140608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13906880005120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99242496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13906997859840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117854720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3056660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚703881 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13495463675904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3642318 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 411515571712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59338749135872 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59338390937088 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 _step‚29 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3632692 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1545670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360‰?Ÿ£‚(:” ÍÇìÒ˜Ä⺠gpu.0.gpu‚90.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%O¥°¤‚(:¥ ÍÇìÒ”̦¼! disk.dm-0.out‚101.43994140625 disk.rbd1.out‚0.75 disk.sda2.out‚0 cpu‚26.052507379105016 disk.drbd2165.out‚0 disk.sda1.in‚0 network.recv‚133374$ memory_percent‚96.33514881134033" proc.memory.rssMB‚ 2638.8359375 disk.sda1.out‚0$ disk./.usageGB‚585.6276550292969 disk.dm-0.in‚ 630.90234375 disk.dm-1.out‚ 7.2734375" disk.nvme0c0n1.in‚ 630.90234375& disk.nvme0c0n1.out‚101.43994140625 disk.sda3.in‚0) disk./.usagePercent‚39.30196330334755 disk.dm-1.in‚0 disk.nvme0n1.in‚ 630.90234375 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.out‚ 7.2734375( proc.memory.availableMB‚ 600.44921875$ disk.nvme0n1.out‚101.43994140625 disk.sda3.out‚ 7.2734375* proc.memory.percent‚16.106176376342773 disk.rbd0.out‚0 network.sent‚449456 disk.drbd2165.in‚0 disk.rbd0.in‚0@ #õ0¥+ _wandbruntime‚330  _runtime‚330¨åRº-&¦¶M VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3641744 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚843856 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59079493865472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59079476826112 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 412455071232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59192937376340 trainer/global_step‚399 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58786046197760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59487002799616 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 407508934144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13528911046656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406874139220 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59079476826112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚30 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚705623 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13528892434432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843578381006837e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59079493865472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 F¨À"bDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3641744 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚843856 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59079493865472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59079476826112 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 412455071232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59192937376340 trainer/global_step‚399 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58786046197760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59487002799616 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 407508934144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13528911046656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406874139220 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59079476826112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚30 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚705623 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13528892434432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843578381006837e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59079493865472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 OJýbE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP{ëIÔHþbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP2½ˆJÿbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPç9®ÌJ€bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP`(áô;b6 jet/accuracy_3_of_4"trainer/global_step2:JP5NºbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPŸËQN»bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÇ’ïH¼bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¶þ‘[N½bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP×ÞÆJ¾bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPµ¯ŸJ¿bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPðOHÀbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP­Ú?HÁbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP>‹s8HÂbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÜ2êšHÃbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPi9`hJÄbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP.¿ØNÅbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÏ`ÔI@Æb; particle/accuracy_3_of_3"trainer/global_step2:JP'Ù6$;Çb6 loss/val_total_loss"trainer/global_step2:JPÈCïHÈbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP¦B‚HÉbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPü¹öJÊbE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÖÒ=,NËbI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP/.»÷8Ìb3 particle/f_score"trainer/global_step2:JP\Vo¼0Í+ _wandbruntime‚348  _runtime‚348koe =Î…z & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0729166716337204 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3262077569961548 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10661890357732773 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.10841178894042969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26905399560928345 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚32 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11241984367370605 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928755760192871 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1192561462521553 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.17401619255542755 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.19168159365653992 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9805203676223755 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002080228179693222 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.313154399394989 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0714285746216774 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.09701436012983322 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06964991241693497 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.03343592584133148 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.05115540325641632 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 /particle/accuracy_3_of_3‚0.9953175783157349 (loss/val_total_loss‚24.8052525524565 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10041667520999908 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.28827205300331116 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.47070902585983276 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964293241500854 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.060449741780757904 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04137270152568817 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15133869647979736 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061528876423835754 jet/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.08481957763433456 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.760995090007782 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08827933669090271 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.13109368085861206 particle/accuracy_1_of_4‚1 +particle/specificity‚0.8545006513595581 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2036527842283249 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.03542128950357437 *jet/accuracy_1_of_4‚0.4432622492313385 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7691445350646973 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925285577774048 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.12412229180335999 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3888889253139496 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9199994206428528 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.08239623159170151 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.03750000149011612 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.989777684211731 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.997619092464447 jet/accuracy_4_of_4‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6698717474937439 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.06872136890888214 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.3860432207584381 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9499011039733887 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_1_of_1‚0.08443712443113327 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.09686292707920074 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.032831061631441116 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,particle/sensitivity‚0.47156229615211487 ;"Purity/*FRt1SRqqt0FBt/event_purity‚0.047796111553907394 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11436069756746292 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.002367651090025902 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.10105820745229721 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.12340161949396133 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.08239623159170151 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9982309937477112 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843578567291112e+09 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8943982720375061 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.17576338350772858 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_5_of_5‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.02609671838581562 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.09445834904909134 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.034128520637750626 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.004085172433406115 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0020453673787415028 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9749504923820496 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004766121506690979 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11049038171768188 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.026470597833395004 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.025206735357642174 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14962390065193176 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.07823167741298676 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9905806183815002 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0022125346586108208 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.04757744073867798 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.1749170571565628 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99139803647995 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.556746244430542 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984673857688904 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2626011073589325 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_3‚0.544662356376648 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11831842362880707 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1299697905778885 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 +jet/accuracy_2_of_4‚0.07680059969425201 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990688681602478 & Purity/2FRt1SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.1213252991437912 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07136780023574829 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.08897270262241364 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.10519550740718842 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1766102910041809 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.11448424309492111 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940020442008972 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09961330145597458 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2829766571521759 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.11957024037837982 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.03535393252968788 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.974076509475708 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16186180710792542 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6validation_average_jet_accuracy‚0.1220199391245842 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7158495783805847 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.004648224450647831 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1894015073776245 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0022668298333883286 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061528876423835754 _runtime‚ 348.805228081 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.30560192465782166 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14058616757392883 +jet/accuracy_2_of_3‚0.12317240983247757 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.08681090921163559 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.10013483464717865 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.13109368085861206 (particle/accuracy‚0.7722594141960144 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970133900642395 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3781863749027252 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.10251682996749878 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44999998807907104 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.09939239174127579 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.14759571850299835 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004573770798742771 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9987747669219971 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3332604765892029 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9807051420211792 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882646203041077 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11181765794754028 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2986111044883728 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09639633446931839 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.37498489022254944 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928761720657349 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.07440205663442612 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.12842027842998505 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00449477368965745 ,jet/accuracy_3_of_4‚0.011249065399169922 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2578125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.10922477394342422 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05376039817929268 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0360824391245842 ,jet/accuracy_3_of_3‚0.008169857785105705 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0625 trainer/global_step‚399 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9987747669219971 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.027527902275323868 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.10696578025817871 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.09692885726690292 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.08642243593931198 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15162721276283264 particle/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.03719170764088631 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.034128520637750626„ðmü5ωz :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04137270152568817 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15133869647979736 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061528876423835754 jet/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.08481957763433456 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.760995090007782 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08827933669090271 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.13109368085861206 particle/accuracy_1_of_4‚1 +particle/specificity‚0.8545006513595581 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2036527842283249 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.03542128950357437 *jet/accuracy_1_of_4‚0.4432622492313385 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7691445350646973 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925285577774048 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.12412229180335999 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3888889253139496 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9199994206428528 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.08239623159170151 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.03750000149011612 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.989777684211731 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.997619092464447 jet/accuracy_4_of_4‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6698717474937439 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.06872136890888214 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.3860432207584381 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9499011039733887 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_1_of_1‚0.08443712443113327 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.09686292707920074 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.032831061631441116 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,particle/sensitivity‚0.47156229615211487 ;"Purity/*FRt1SRqqt0FBt/event_purity‚0.047796111553907394 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11436069756746292 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.002367651090025902 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.10105820745229721 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.12340161949396133 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.08239623159170151 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9982309937477112 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843578567291112e+09 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8943982720375061 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.17576338350772858 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_5_of_5‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.02609671838581562 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.09445834904909134 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.034128520637750626 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.004085172433406115 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0020453673787415028 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9749504923820496 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004766121506690979 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11049038171768188 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.026470597833395004 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.025206735357642174 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14962390065193176 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.07823167741298676 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9905806183815002 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0022125346586108208 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.04757744073867798 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.1749170571565628 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99139803647995 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.556746244430542 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984673857688904 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2626011073589325 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_3‚0.544662356376648 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11831842362880707 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1299697905778885 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 +jet/accuracy_2_of_4‚0.07680059969425201 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990688681602478 & Purity/2FRt1SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.1213252991437912 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07136780023574829 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.08897270262241364 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.10519550740718842 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1766102910041809 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.11448424309492111 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940020442008972 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09961330145597458 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2829766571521759 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.11957024037837982 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.03535393252968788 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.974076509475708 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16186180710792542 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6validation_average_jet_accuracy‚0.1220199391245842 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7158495783805847 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.004648224450647831 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1894015073776245 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0022668298333883286 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061528876423835754 _runtime‚ 348.805228081 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.30560192465782166 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14058616757392883 +jet/accuracy_2_of_3‚0.12317240983247757 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.08681090921163559 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.10013483464717865 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.13109368085861206 (particle/accuracy‚0.7722594141960144 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970133900642395 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3781863749027252 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.10251682996749878 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44999998807907104 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.09939239174127579 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.14759571850299835 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004573770798742771 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9987747669219971 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3332604765892029 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9807051420211792 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882646203041077 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11181765794754028 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2986111044883728 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09639633446931839 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.37498489022254944 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928761720657349 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.07440205663442612 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.12842027842998505 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00449477368965745 ,jet/accuracy_3_of_4‚0.011249065399169922 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2578125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.10922477394342422 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05376039817929268 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0360824391245842 ,jet/accuracy_3_of_3‚0.008169857785105705 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0625 trainer/global_step‚399 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9987747669219971 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.027527902275323868 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.10696578025817871 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.09692885726690292 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.08642243593931198 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15162721276283264 particle/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.03719170764088631 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.034128520637750626 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0729166716337204 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3262077569961548 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10661890357732773 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.10841178894042969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26905399560928345 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚32 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11241984367370605 ?&Purity/2FRt2SRqqt0F•'6ýBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928755760192871 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1192561462521553 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.17401619255542755 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.19168159365653992 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9805203676223755 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002080228179693222 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.313154399394989 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0714285746216774 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.09701436012983322 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06964991241693497 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.03343592584133148 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.05115540325641632 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 /particle/accuracy_3_of_3‚0.9953175783157349 (loss/val_total_loss‚24.8052525524565 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10041667520999908 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.28827205300331116 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.47070902585983276 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964293241500854 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.060449741780757904 gGšð”Ђ(:‰ ëÇìÒ „¼ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kŸQX£Ñ‚(:˜ ëÇìҬק¾# disk./.usageGB‚586.107837677002! disk.dm-0.out‚355.22119140625 disk.dm-1.in‚0 network.recv‚142320 disk.sda1.in‚0! proc.memory.rssMB‚ 2719.546875$ disk.nvme0n1.out‚355.22119140625 disk.drbd2165.out‚0 disk.dm-1.out‚7.9375 disk.nvme0n1.in‚ 799.9296875 disk.sda2.out‚0 disk.dm-0.in‚ 799.9296875* proc.memory.percent‚16.598796844482422 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0$ memory_percent‚96.96965217590332 disk.rbd1.out‚0.8125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 cpu‚26.170406934141113 proc.cpu.threads‚17) disk./.usagePercent‚39.33418876373511 disk.sda.out‚7.9375 disk.sda3.out‚7.9375 disk.rbd1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 799.9296875& disk.nvme0c0n1.out‚355.22119140625' proc.memory.availableMB‚ 496.4921875 network.sent‚523309\+‚¬”Ò‚(:‰ úÇìÒ…¹Ü» gpu.0.gpu‚80.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LRqŠ¥Ó‚(:š úÇìҾϢ¼ network.recv‚146967 disk.rbd1.in‚0 disk.rbd1.out‚0.8125 disk.rbd0.out‚0 disk.dm-0.out‚ 378.494140625! proc.memory.rssMB‚ 2719.546875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 8.4765625 network.sent‚580491) disk./.usagePercent‚39.33413295409265 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 378.494140625 disk.nvme0n1.in‚ 799.9296875 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 8.4765625 disk.sda3.in‚0 disk.dm-0.in‚ 799.9296875 disk.sda1.out‚0$ memory_percent‚96.96931838989258! disk.nvme0c0n1.in‚ 799.9296875 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 8.4765625 disk.sda1.in‚0 cpu‚26.161781482844507# disk./.usageGB‚586.107006072998" disk.nvme0n1.out‚ 378.494140625& proc.memory.availableMB‚ 496.546875* proc.memory.percent‚16.5987968444824223±É°ŸÔ‚(:” ‰ÈìÒŸ§þº gpu.0.gpu‚89.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚225.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.195800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÍ ³Õ‚(:¨ ‰ÈìÒËžý¼ network.sent‚585786* disk./.usagePercent‚39.334118361663194 disk.rbd0.out‚0 disk.nvme0n1.in‚ 799.9296875 disk.sda.out‚ 8.78515625 disk.drbd2165.out‚0 network.recv‚150677 disk.sda1.out‚0$ memory_percent‚96.97296619415283$ disk./.usageGB‚586.1067886352539 disk.rbd1.out‚0.875 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0! disk.dm-0.out‚606.57373046875# proc.memory.rssMB‚ 2719.55078125* proc.memory.percent‚16.598820686340332 disk.rbd0.in‚0$ disk.nvme0n1.out‚606.57373046875 disk.sda.in‚0 disk.dm-1.out‚ 8.78515625 cpu‚26.152814846072218 proc.cpu.threads‚17 disk.dm-0.in‚ 799.9296875& disk.nvme0c0n1.out‚606.57373046875 disk.sda3.out‚ 8.78515625 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 799.9296875 disk.sda2.out‚0( proc.memory.availableMB‚ 495.94921875 disk.sda3.in‚0 disk.dm-1.in‚0Õè‹0Ö+ _wandbruntime‚391  _runtime‚391üÕCÄ×¾ trainer/global_step‚449 % _timestamp‚1.7843578990667348e+09 _runtime‚ 391.142104195 _step‚33 & lr-AdamW/pg1‚0.0004349930678280865 & lr-AdamW/pg2‚0.00043499306782808658ŸáäÈØ & lr-AdamW/pg1‚0.0004349930678280865 & lr-AdamW/pg2‚0.0004349930678280865 trainer/global_step‚449 % _timestamp‚1.7843578990667348e+09 _runtime‚ 391.142104195 _step‚33!‰¿G0Ù+ _wandbruntime‚391  _runtime‚391ââÄ °'ÚªO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68223870967808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68347591081569 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473858748001 _runtime‚ 391.143376536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68698437628416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67873749372928 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4167152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15182832314368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 474583699968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68223870967808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1770032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 474199297883 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558926 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68347948670811 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4157177 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4157177 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68698796005888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15663504554496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1769776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 # _timestamp‚1.78435789906786e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚976128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚449 _step‚34 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480690852352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¶^3´'Û®O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68223870967808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68347591081569 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473858748001 _runtime‚ 391.143376536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68698437628416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67873749372928 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4167152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15182832314368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 474583699968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68223870967808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1770032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 474199297883 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558926 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68347948670811 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4157177 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4157177 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68698796005888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15663504554496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1769776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 # _timestamp‚1.78435789906786e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚976128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚449 _step‚34 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480690852352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0"µ©SP0Ü+ _wandbruntime‚391  _runtime‚391>nñòBݵM WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 475596326912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 475596326912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚86211584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 $ _timestamp‚1.784357899679307e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68371094958080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 474869801723 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15216242460672 trainer/global_step‚449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 481741648384 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68371111997440 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3566558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7732787 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 475210351717 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 391.755041439 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚795393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68371111997440 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4166229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437¯Ùþ¿&Þ¹M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68371111997440 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4166229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 475596326912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 475596326912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚86211584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 $ _timestamp‚1.784357899679307e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68371094958080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 474869801723 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15216242460672 trainer/global_step‚449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 481741648384 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68371111997440 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3566558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7732787 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 475210351717 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 391.755041439 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚795393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616#´Øë0ß+ _wandbruntime‚391  _runtime‚391{¿Ûàû 0loss/FRt1/detection_loss‚0.49237357927732384 1loss/SRqqt2/detection_loss‚0.7461707593104276 trainer/global_step‚449 % _timestamp‚1.7843578996816537e+09 2loss/SRqqt2/assignment_loss‚1.4153713050859653 _runtime‚ 391.756708695 _step‚36 0loss/FRt2/assignment_loss‚3.0516718725652194 0loss/SRqqt1/assignment_loss‚1.50938789615183 /loss/FRt2/detection_loss‚1.4633182918697183 /loss/FBt2/detection_loss‚2.7848892406719035 0loss/FRt1/assignment_loss‚3.8543906240928463 0loss/FBt1/assignment_loss‚0.5948648310292055 1loss/FBt2/assignment_loss‚0.18407414702747235 0loss/SRqqt1/detection_loss‚2.390112346666336 /loss/FBt1/detection_loss‚10.075663643639261 %loss/total_loss‚28.56228853738751 epoch‚1übF'áÿ 0loss/FRt1/assignment_loss‚3.8543906240928463 0loss/FBt1/assignment_loss‚0.5948648310292055 1loss/FBt2/assignment_loss‚0.18407414702747235 0loss/SRqqt1/detection_loss‚2.390112346666336 /loss/FBt1/detection_loss‚10.075663643639261 %loss/total_loss‚28.56228853738751 epoch‚1 0loss/FRt1/detection_loss‚0.49237357927732384 1loss/SRqqt2/detection_loss‚0.7461707593104276 trainer/global_step‚449 % _timestamp‚1.7843578996816537e+09 2loss/SRqqt2/assignment_loss‚1.4153713050859653 _runtime‚ 391.756708695 _step‚36 0loss/FRt2/assignment_loss‚3.0516718725652194 0loss/SRqqt1/assignment_loss‚1.50938789615183 /loss/FRt2/detection_loss‚1.4633182918697183 /loss/FBt2/detection_loss‚2.7848892406719035$´¨ÐP”â‚(:‰ ˜ÈìÒŠê–» gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚182.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05-Âq«ã‚(:  ˜ÈìҊݼ disk.sda.in‚0 disk.sda3.out‚ 9.06640625' proc.memory.availableMB‚ 495.9453125 disk.dm-1.out‚ 9.06640625 network.recv‚154214) disk./.usagePercent‚39.33413116203991 disk.sda2.in‚0 disk.dm-0.in‚ 799.9296875 disk.drbd2165.out‚0 disk.dm-0.out‚ 610.857421875# proc.memory.rssMB‚ 2719.55078125 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.875 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚590949$ disk./.usageGB‚586.1069793701172 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.598820686340332 disk.sda1.out‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 610.857421875" disk.nvme0n1.out‚ 610.857421875 cpu‚26.148671998500653 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 799.9296875 disk.nvme0n1.in‚ 799.9296875 disk.sda.out‚ 9.06640625$ memory_percent‚96.97299003601074Jk‰àœä‚(:‘ §ÈìÒÕ’¬½ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n©X§°å‚(:¥ §ÈìÒ‹ÚÕ½ disk.dm-0.in‚ 942.74609375 disk.rbd1.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚613.9091796875 disk.sda2.out‚0 disk.drbd2165.out‚0 network.recv‚159068* disk./.usagePercent‚39.334133466107716 disk.rbd1.out‚0.9375 disk.rbd0.in‚0 disk.sda.out‚ 9.3984375 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚96.97439670562744" disk.nvme0c0n1.in‚ 942.74609375 disk.sda1.in‚0* proc.memory.percent‚16.598820686340332 disk.nvme0n1.in‚ 942.74609375 disk.sda.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 495.71484375# proc.memory.rssMB‚ 2719.55078125$ disk./.usageGB‚586.1070137023926 disk.sda2.in‚0 disk.dm-0.out‚613.9091796875# disk.nvme0n1.out‚613.9091796875 disk.dm-1.out‚ 9.3984375 cpu‚26.14072781389913 proc.cpu.threads‚17 network.sent‚653606 disk.sda3.out‚ 9.3984375ƒË·Ý0æ+ _wandbruntime‚431  _runtime‚431I ³EÄç¾ _step‚37 & lr-AdamW/pg1‚0.0004349723932326992 & lr-AdamW/pg2‚0.0004349723932326992 trainer/global_step‚499 % _timestamp‚1.7843579398717587e+09 _runtime‚ 431.950211795÷‡+Èè & lr-AdamW/pg1‚0.0004349723932326992 & lr-AdamW/pg2‚0.0004349723932326992 trainer/global_step‚499 % _timestamp‚1.7843579398717587e+09 _runtime‚ 431.950211795 _step‚37%Sªƒå0é+ _wandbruntime‚431  _runtime‚431ˆÁ£Nµ'ê¯O P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1960659 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75199265927168 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17386332474368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4621604 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 532167326720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚880898 _runtime‚ 431.950356308 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17386437567488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16854251628544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4621604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 525279201792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 524815784791 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75585922449408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76111184611840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 525279201792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76111542989312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75723741162077 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3940528 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3940526 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4611629 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 525620539904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4611629 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚880901 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1079761 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75585922449408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1080037 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 524475234909 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 % _timestamp‚1.7843579398725548e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75585905410048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚38 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940526 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 532080845824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17386332474368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4621604 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 532167326720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚880898 _runtime‚ 431.950356308 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17386437567488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16854251628544 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4621604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 525279201792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 524815784791 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75585922449408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76111184611840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 525279201792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚a4;­¼ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76111542989312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75723741162077 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3940528 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3940526 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4611629 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 525620539904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4611629 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚880901 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1079761 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75585922449408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1080037 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 524475234909 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 % _timestamp‚1.7843579398725548e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75585905410048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚38 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940528 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940526 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 532080845824 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚39 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75733163479040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76259438268416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3948158 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3948160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚882643 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 533119034880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 533195022848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1082107 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75733146439680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75733146439680 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4630670 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17420799421952 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3948158 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75733163479040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 525486288631 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75345793638400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 526291828736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75871620477025 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 525826838625 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4620681 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4620681 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784357940489324e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1081866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75871262887671 trainer/global_step‚499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75987968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94600192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3948160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚882643 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 533119034880 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 533195022848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1082107 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75733146439680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75733146439680 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4630670 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17420799421952 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3948158 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75733163479040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 525486288631 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75345793638400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 526291828736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75871620477025 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 525826838625 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4620681 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4620681 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784357940489324e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1081866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75871262887671 trainer/global_step‚499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75987968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94600192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚39 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75733163479040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76259438268416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0'_@»s0ï+ _wandbruntime‚432  _runtime‚432cÓ+üðö 1loss/SRqqt1/assignment_loss‚1.626419731856961 1loss/FBt2/assignment_loss‚0.06007174045764191 _step‚40 /loss/FRt1/assignment_loss‚3.843815623736772 /loss/FRt2/assignment_loss‚2.895318801496543 epoch‚1 2loss/SRqqt2/assignment_loss‚1.4572727742734333 /loss/FRt2/detection_loss‚1.5291725736742177 0loss/SRqqt1/detection_loss‚2.288341484988638 1loss/SRqqt2/detection_loss‚0.7869266813721401 .loss/FBt2/detection_loss‚3.360103130414832 &loss/total_loss‚29.979679105175812 trainer/global_step‚499 0loss/FBt1/assignment_loss‚0.5081035692135225 -loss/FRt1/detection_loss‚0.52781226920344 .loss/FBt1/detection_loss‚11.09632072448767 % _timestamp‚1.7843579404916685e+09 _runtime‚ 432.566732975›i5Ýñú trainer/global_step‚499 0loss/FBt1/assignment_loss‚0.5081035692135225 -loss/FRt1/detection_loss‚0.52781226920344 .loss/FBt1/detection_loss‚11.09632072448767 % _timestamp‚1.7843579404916685e+09 _runtime‚ 432.566732975 1loss/SRqqt1/assignment_loss‚1.626419731856961 1loss/FBt2/assignment_loss‚0.06007174045764191 _step‚40 /loss/FRt1/assignment_loss‚3.843815623736772 /loss/FRt2/assignment_loss‚2.895318801496543 epoch‚1 2loss/SRqqt2/assignment_loss‚1.4572727742734333 /loss/FRt2/detection_loss‚1.5291725736742177 0loss/SRqqt1/detection_loss‚2.288341484988638 1loss/SRqqt2/detection_loss‚0.7869266813721401 .loss/FBt2/detection_loss‚3.360103130414832 &loss/total_loss‚29.979679105175812(õÖ³ò‚(:¨ ¶ÈìÒèü™¼ disk.nvme0n1.in‚ 942.74609375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 942.74609375 disk.rbd0.out‚0 disk.dm-1.out‚ 9.84765625( proc.memory.availableMB‚ 399.84765625$ disk./.usageGB‚586.1070861816406 disk.sda3.in‚0 disk.sda3.out‚ 9.84765625# disk.nvme0n1.out‚618.3857421875# memory_percent‚97.5595235824585 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.out‚618.3857421875 disk.rbd1.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚618.3857421875# proc.memory.rssMB‚ 2719.55078125 disk.sda.out‚ 9.84765625 disk.dm-1.in‚0 disk.rbd1.out‚0.9375" disk.nvme0c0n1.in‚ 942.74609375 network.recv‚162739* proc.memory.percent‚16.598820686340332 proc.cpu.threads‚17 network.sent‚658882* disk./.usagePercent‚39.334138330250866 disk.sda2.out‚0 cpu‚26.138480511088176 8 i’ó‚(:‡ ¶ÈìÒ¬¿¦½ gpu.0.gpu‚95.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÍú”ô‚(:‰ ÅÈìÒÑ­”¼ gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ÖË™¨õ‚(: ÅÈìÒ–œ–½ disk.sda3.out‚ 10.0546875 disk.drbd2165.out‚0 disk.dm-1.out‚ 10.0546875 disk.rbd0.in‚0 network.sent‚721534# disk./.usageGB‚586.107105255127 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0* proc.memory.percent‚16.598844528198242 disk.sda.out‚ 10.0546875 disk.nvme0n1.in‚ 942.74609375" disk.nvme0n1.out‚ 622.548828125" proc.memory.rssMB‚ 2719.5546875 cpu‚26.130300808247426 disk.dm-0.in‚ 942.74609375$ disk.nvme0c0n1.out‚ 622.548828125 disk.sda1.in‚0' proc.memory.availableMB‚ 10.51953125) disk./.usagePercent‚39.33413961028854 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.out‚ 622.548828125" disk.nvme0c0n1.in‚ 942.74609375 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚99.93579387664795 proc.cpu.threads‚17 network.recv‚167554 disk.rbd1.out‚1~iF”ö‚(:‰ ÔÈìÒ›ˆ¼ gpu.0.gpu‚41.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0——´W±÷‚(:¦ ÔÈìңɼ$ disk./.usageGB‚586.1081199645996 disk.dm-0.in‚ 942.74609375! disk.dm-0.out‚628.92041015625 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 10.35546875 disk.sda3.in‚0& proc.memory.availableMB‚ 10.0234375* disk./.usagePercent‚39.334207708292645 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 942.74609375 network.recv‚171085 disk.sda1.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚628.92041015625$ disk.nvme0n1.out‚628.92041015625 disk.dm-1.out‚ 10.35546875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 942.74609375 disk.sda3.out‚ 10.35546875* proc.memory.percent‚16.598844528198242 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.93882179260254" proc.memory.rssMB‚ 2719.5546875 network.sent‚726884 disk.rbd1.out‚1 disk.rbd0.out‚0 cpu‚26.12841694699051 proc.cpu.threads‚17mñ½0ø+ _wandbruntime‚472  _runtime‚472up]nÆùÀ _runtime‚ 472.972076866 _step‚41 ' lr-AdamW/pg1‚0.00043493775078468257 ' lr-AdamW/pg2‚0.00043493775078468257 trainer/global_step‚549 % _timestamp‚1.7843579808967915e+09‡€^ÊúÄ trainer/global_step‚549 % _timestamp‚1.7843579808967915e+09 _runtime‚ 472.972076866 _step‚41 ' lr-AdamW/pg1‚0.00043493775078468257 ' lr-AdamW/pg2‚0.00043493775078468257)ÂìŠ0û+ _wandbruntime‚472  _runtime‚472ÆT‡Ñ6ü¯O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82947973931008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83099891242585 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.#|6&all.freed‚83523931595264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1183121 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19109024729088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82947956891648 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 583421655040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5076056 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83524289972736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1183382 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82947956891648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19109129822208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4322128 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322126 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚968151 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18525708167168 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82524799520768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83523931595264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 576316041728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚968148 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5076056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2151533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82524816560128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _step‚42 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83524289972736 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9398184 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2151269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚549 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 576316041728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 % _timestamp‚1.7843579808975968e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 575974703616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 575974703616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82947973931008 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 575432271699 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322128 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5066081 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 576316041728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚968148 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5076056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2151533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82524816560128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _step‚42 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83524289972736 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9398184 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2151269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚549 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 576316041728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 % _timestamp‚1.7843579808975968e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 575974703616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 575974703616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82947973931008 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 575432271699 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322128 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5066081 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83523931595264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1183121 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19109024729088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82947956891648 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 583421655040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5076056 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83524289972736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1183382 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616*¼|ƒ±0þ+ _wandbruntime‚473  _runtime‚473,XóB¿&ÿ¹M ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18559118313472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚969896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1185203 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4329758 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1185454 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83095197921280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 577328675840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5075133 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83095197921280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83247770557533 _runtime‚ 473.588650057 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2155350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18559136925696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚43 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚969893 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5085122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83672185251840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329760 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 576443325533 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843579815130754e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83672543636480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329758 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 576102775539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 584447200768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83247412968179 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83095214960640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5075133@¶ €½M J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5075133 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18559118313472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚969896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1185203 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4329758 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1185454 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83095197921280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 577328675840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5075133 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83095197921280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83247770557533 _runtime‚ 473.588650057 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2155350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18559136925696 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚43 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚969893 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5085122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83672185251840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329760 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 576443325533 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843579815130754e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83672543636480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329758 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 576102775539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 584447200768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83247412968179 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83095214960640+²îî0+ _wandbruntime‚473  _runtime‚473O JËÿ‚ù /loss/FRt2/detection_loss‚1.3867122939834513 .loss/FBt2/detection_loss‚7.763450865169034 trainer/global_step‚549 _step‚44 1loss/SRqqt1/assignment_loss‚1.319766498210256 0loss/FBt1/assignment_loss‚0.7890021501042481 1loss/FBt2/assignment_loss‚0.11096879546759685 1loss/SRqqt1/detection_loss‚1.9997559271247711 -loss/FBt1/detection_loss‚4.85652449131012 epoch‚1 0loss/FRt1/assignment_loss‚3.6999890083490414 /loss/FRt2/assignment_loss‚2.800547398060651 /loss/FRt1/detection_loss‚0.5803941428797995 1loss/SRqqt2/detection_loss‚0.7917587144294761 &loss/total_loss‚27.466011009594432 _runtime‚ 473.590540275 2loss/SRqqt2/assignment_loss‚1.3671407245059861 % _timestamp‚1.7843579815154967e+09œ$Õƒý epoch‚1 0loss/FRt1/assignment_loss‚3.6999890083490414 /loss/FRt2/assignment_loss‚2.800547398060651 /loss/FRt1/detection_loss‚0.5803941428797995 1loss/SRqqt2/detection_loss‚0.7917587144294761 &loss/total_loss‚27.466011009594432 _runtime‚ 473.590540275 2loss/SRqqt2/assignment_loss‚1.3671407245059861 % _timestamp‚1.7843579815154967e+09 /loss/FRt2/detection_loss‚1.3867122939834513 .loss/FBt2/detection_loss‚7.763450865169034 trainer/global_step‚549 _step‚44 1loss/SRqqt1/assignment_loss‚1.319766498210256 0loss/FBt1/assignment_loss‚0.7890021501042481 1loss/FBt2/assignment_loss‚0.11096879546759685 1loss/SRqqt1/detection_loss‚1.9997559271247711 -loss/FBt1/detection_loss‚4.85652449131012,¥F“„‚(:ˆ ãÈìÒ¯ùº gpu.0.gpu‚87.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‡\·ª…‚(:Ÿ ãÈìÒ±Òê¼ disk.sda2.out‚0! disk.dm-0.out‚632.42724609375 disk.rbd1.in‚0" proc.memory.rssMB‚ 2719.5546875 network.sent‚732285) disk./.usagePercent‚39.33419900403648 disk.dm-1.in‚0 disk.rbd1.out‚1.0625 disk.nvme0n1.in‚ 980.47265625 disk.sda3.out‚10.5625 cpu‚26.124074774926473 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚10.5625 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme0c0n1.out‚632.42724609375& proc.memory.availableMB‚ 9.98046875* proc.memory.percent‚16.598844528198242 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚99.93908405303955 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 980.47265625 disk.sda1.out‚0$ disk./.usageGB‚586.1079902648926 disk.dm-0.in‚ 980.47265625 disk.dm-1.out‚10.5625$ disk.nvme0n1.out‚632.42724609375 disk.sda3.in‚0 disk.sda1.in‚0 network.recv‚174937aâFN”†‚(:‰ òÈìÒå”Õ» gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚259.652' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~€›:µ‡‚(:ª òÈìÒÙø¼# disk.nvme0n1.out‚637.3935546875 disk.dm-1.out‚ 11.05859375# memory_percent‚99.9403715133667 disk.sda.out‚ 11.05859375 disk.sda3.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.598868370056152! disk.nvme0n1.in‚ 1079.33984375 disk.sda3.out‚ 11.05859375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚637.3935546875& proc.memory.availableMB‚ 9.76953125# proc.memory.rssMB‚ 2719.55859375 proc.cpu.threads‚17 network.recv‚179420) disk./.usagePercent‚39.33420156411182 disk.drbd2165.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚637.3935546875$ disk./.usageGB‚586.1080284118652 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1079.33984375 disk.dm-0.in‚ 1079.33984375 cpu‚26.12336672395127 network.sent‚794986 disk.rbd1.in‚0 disk.rbd1.out‚1.0625lRvz“ˆ‚(:ˆ ÉìÒÓÔ¶º gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚250.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÕ\Ï¥‰‚(:š ÉìÒ¯¦¿¼ disk.drbd2165.in‚0 disk.dm-0.in‚ 1079.33984375 disk.dm-1.in‚0 disk.rbd1.out‚1.125* proc.memory.percent‚16.598868370056152 disk.rbd1.in‚0# proc.memory.rssMB‚ 2719.55859375! disk.nvme0c0n1.out‚ 640.046875 disk.sda.in‚0 disk.dm-1.out‚ 11.2890625 disk.dm-0.out‚ 640.046875# memory_percent‚99.9413251876831# disk./.usageGB‚586.107967376709 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.out‚ 640.046875 disk.sda.out‚ 11.2890625 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1079.33984375 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 cpu‚26.118078722694158 proc.cpu.threads‚17 network.sent‚800389 disk.sda3.out‚ 11.2890625 network.recv‚183510 disk.sda3.in‚0& proc.memory.availableMB‚ 9.61328125) disk./.usagePercent‚39.33419746799128! disk.nvme0n1.in‚ 1079.33984375 disk.sda1.out‚0MwÕä0Š+ _wandbruntime‚513  _runtime‚513³+Ä‹¾ & lr-AdamW/pg2‚0.0004348891427090562 trainer/global_step‚599 % _timestamp‚1.7843580216217656e+09 _runtime‚ 513.697118207 _step‚45 & lr-AdamW/pg1‚0.0004348891427090562g±{Ȍ & lr-AdamW/pg1‚0.0004348891427090562 & lr-AdamW/pg2‚0.0004348891427090562 trainer/global_step‚599 % _timestamp‚1.7843580216217656e+09 _runtime‚ 513.697118207 _step‚45-7AË!0+ _wandbruntime‚513  _runtime‚5136ú¨òº'Ž´O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89850333114368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90310008373248 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 627011543552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20197146093568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20831616496640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626670205440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111384576 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634489015296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90310008373248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5520533 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90476041323093 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2341942 $ _timestamp‚1.784358021622572e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626670205440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90310025412608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1055398 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1286279 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92772352 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1286541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4703726 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90937036956160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90310025412608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 627011543552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 _runtime‚ 513.698046365 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚599 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90936678578688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90310025412608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1055398 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1286279 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92772352 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1286541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4703726 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90937036956160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90310025412608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 627011543552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 _runtime‚ 513.698046365 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚599 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90936678578688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20197146093568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20831616496640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 JADeçcBzÑviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626670205440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111384576 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099.Ðr>0+ _wandbruntime‚514  _runtime‚514Yž³öÄ&‘¾M QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 635597559808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89996860825600 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 627059812441 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 627682832384 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91085290619904 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90457249402880 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 627682832384 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90623563048687 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20230574852096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4711360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90457266442240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1288330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1057146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4711360 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2345740 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 626719262447 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90457266442240 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90623920638041 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 635508988928 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570880 $ _timestamp‚1.784358022238534e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5539574 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5539574 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 628024177664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 _step‚47 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1057143 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570880 $ _timestamp‚1.784358022238534e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5539574 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5539574 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 628024177664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 _step‚47 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1057143 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90457266442240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1288330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1057146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4711360 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2345740 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 626719262447 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90457266442240 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90623920638041 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508/œù…0“+ _wandbruntime‚514  _runtime‚514Ñð¨”ü 0loss/FRt2/assignment_loss‚2.5246373904921646 2loss/SRqqt1/assignment_loss‚1.4987573316902854 2loss/SRqqt2/assignment_loss‚1.3614998464255572 1loss/FBt2/assignment_loss‚0.10161109060985983 /loss/FRt2/detection_loss‚1.5065205549548524 1loss/SRqqt1/detection_loss‚1.8399226546171121 1loss/SRqqt2/detection_loss‚0.7112858160919747 trainer/global_step‚599 /loss/FRt1/detection_loss‚0.6267576691162604 epoch‚1 _runtime‚ 514.316429051 _step‚48 /loss/FBt2/detection_loss‚3.3345658573138133 &loss/total_loss‚20.349308386169568 /loss/FBt1/assignment_loss‚0.509913493619961 % _timestamp‚1.7843580222413445e+09 /loss/FRt1/assignment_loss‚3.444596307973067 /loss/FBt1/detection_loss‚2.8892403732646597ßX<•€ _step‚48 /loss/FBt2/detection_loss‚3.3345658573138133 &loss/total_loss‚20.349308386169568 /loss/FBt1/assignment_loss‚0.509913493619961 % _timestamp‚1.7843580222413445e+09 /loss/FRt1/assignment_loss‚3.444596307973067 /loss/FBt1/detection_loss‚2.8892403732646597 0loss/FRt2/assignment_loss‚2.5246373904921646 2loss/SRqqt1/assignment_loss‚1.4987573316902854 2loss/SRqqt2/assignment_loss‚1.3614998464255572 1loss/FBt2/assignment_loss‚0.10161109060985983 /loss/FRt2/detection_loss‚1.5065205549548524 1loss/SRqqt1/detection_loss‚1.8399226546171121 1loss/SRqqt2/detection_loss‚0.7112858160919747 trainer/global_step‚599 /loss/FRt1/detection_loss‚0.6267576691162604 epoch‚1 _runtime‚ 514.3164290510I7Ô–‚(:‰ ÉìÒ¼˜ˆ» gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q ȹ—‚(:® ÉìÒÛÌá¼ proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-0.in‚ 1079.33984375 cpu‚26.113917047708068 disk.sda3.in‚0! disk.nvme0n1.in‚ 1079.33984375 network.sent‚805699 disk.sda2.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 9.39453125# proc.memory.rssMB‚ 2719.55859375 disk.sda3.out‚ 11.59765625 disk.rbd1.out‚1.125# disk.nvme0c0n1.in‚ 1079.33984375& disk.nvme0c0n1.out‚643.24755859375$ disk.nvme0n1.out‚643.24755859375$ disk./.usageGB‚586.1084442138672 disk.drbd2165.out‚0 disk.dm-1.out‚ 11.59765625 disk.rbd1.in‚0$ memory_percent‚99.94266033172607 network.recv‚187334 disk.sda1.in‚0 disk.sda.in‚0) disk./.usagePercent‚39.33422946893306 disk.sda2.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.598868370056152 disk.sda1.out‚0! disk.dm-0.out‚643.24755859375 disk.dm-1.in‚0 disk.sda.out‚ 11.59765625çw”Š”˜‚(:‰ ŸÉìÒ¾ç» gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º}œÙ»™‚(:° ŸÉìÒšœ½ disk.dm-1.out‚ 12.13671875$ memory_percent‚99.94089603424072* proc.memory.percent‚16.598868370056152 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 1079.33984375& proc.memory.availableMB‚ 9.68359375 network.sent‚868521 network.recv‚192217* disk./.usagePercent‚39.334217948594016 cpu‚26.113502864825154$ disk.nvme0n1.out‚647.20556640625 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2719.55859375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.out‚ 12.13671875 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1079.33984375 disk.sda3.out‚ 12.13671875 disk.sda1.in‚0$ disk./.usageGB‚586.1082725524902 disk.rbd1.out‚1.1875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1079.33984375& disk.nvme0c0n1.out‚647.20556640625! disk.dm-0.out‚647.20556640625íî+;0š+ _wandbruntime‚554  _runtime‚5547æRàÄ›¾ _step‚49 & lr-AdamW/pg1‚0.0004348265721278261 & lr-AdamW/pg2‚0.0004348265721278261 trainer/global_step‚649 % _timestamp‚1.7843580625526972e+09 _runtime‚ 554.631203486çöoÀȜ % _timestamp‚1.7843580625526972e+09 _runtime‚ 554.631203486 _step‚49 & lr-AdamW/pg1‚0.0004348265721278261 & lr-AdamW/pg2‚0.0004348265721278261 trainer/global_step‚6491—Ø5þ0+ _wandbruntime‚554  _runtime‚554¯Š¹ÇCžµO YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 677365707264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22554317548032 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97672076894208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21868584019968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97175883747328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98349425562112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11060311 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1389953 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5085326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5974985 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5984960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 677707045376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1142648 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98349783939584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 KBDeviceStatsMonitor.on_train_batch_st0˜ix art/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 685647047168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21868565407744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21868565407744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97672076894208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21868584019968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97175883747328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98349425562112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11060311 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1389953 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5085326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5974985 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5984960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 677707045376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1142648 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98349783939584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 685647047168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚99772D™GRŸ ‚(:” ®ÉìÒñ«Û» gpu.0.gpu‚66.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚232.707' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.541399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ߶k²¡‚(:§ ®ÉìÒÃœ½$ disk./.usageGB‚586.1083755493164$ memory_percent‚99.94120597839355 network.sent‚872811 disk.drbd2165.out‚0 disk.dm-1.out‚ 12.484375% proc.memory.availableMB‚ 9.6328125 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 1174.03515625 cpu‚26.11014162711303 disk.rbd1.in‚0$ disk.nvme0n1.out‚653.68115234375 disk.sda2.out‚0! disk.dm-0.out‚653.68115234375) disk./.usagePercent‚39.33422486079744 disk.sda.out‚ 12.484375 disk.sda3.in‚0& disk.nvme0c0n1.out‚653.68115234375 disk.sda3.out‚ 12.484375* proc.memory.percent‚16.598868370056152 network.recv‚195033 disk.rbd1.out‚1.1875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1174.03515625! disk.nvme0n1.in‚ 1174.03515625 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2719.55859375 proc.cpu.threads‚17¶òŠ0¢+ _wandbruntime‚555  _runtime‚555X)Å&£¿M VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 686674672640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚649 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5092958 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97819300884480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98498037603328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678719679488 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5994026 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5994026 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5984037 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22588668838912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843580631706274e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5984037 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 678378334208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 677335749355 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1391722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97322377379840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21902012778496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21901994166272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97999713129195 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5092958 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98497679218688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11076995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2536368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98498037603328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1391972 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 555.246254693¿_æR¤ÃM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97999713129195 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5092958 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98497679218688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11076995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2536368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98498037603328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1391972 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 555.246254693 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 686674672640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚649 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5092958 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97819300884480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98498037603328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678719679488 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5994026 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5994026 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5984037 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22588668838912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843580631706274e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5984037 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 TCDeviceStatsMonitor.on_trainBž4w_batch_end/active_bytes.small_pool.freed‚ 678378334208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 677335749355 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1391722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97322377379840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21902012778496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219019941662723ÌÄ”À0¥+ _wandbruntime‚555  _runtime‚555œL¦û 1loss/SRqqt2/detection_loss‚0.7424591667478444 /loss/FBt2/detection_loss‚1.1918660423438128 trainer/global_step‚649 % _timestamp‚1.7843580631732423e+09 0loss/FRt2/assignment_loss‚2.3068227222012707 /loss/FRt2/detection_loss‚1.5240089412301083 /loss/SRqqt1/detection_loss‚1.63471594485319 &loss/total_loss‚16.694386973291877 epoch‚1 _step‚52 2loss/SRqqt1/assignment_loss‚1.2088495914233053 0loss/FBt1/assignment_loss‚0.6124080542939303 /loss/FRt1/assignment_loss‚3.215931856152655 /loss/FRt1/detection_loss‚0.6404820976528639 /loss/FBt1/detection_loss‚2.3266650510556772 _runtime‚ 555.248301299 2loss/SRqqt2/assignment_loss‚1.1424594303810751 1loss/FBt2/assignment_loss‚0.14771807495614658–»"ñ§ÿ 2loss/SRqqt1/assignment_loss‚1.2088495914233053 0loss/FBt1/assignment_loss‚0.6124080542939303 /loss/FRt1/assignment_loss‚3.215931856152655 /loss/FRt1/detection_loss‚0.6404820976528639 /loss/FBt1/detection_loss‚2.3266650510556772 _runtime‚ 555.248301299 2loss/SRqqt2/assignment_loss‚1.1424594303810751 1loss/FBt2/assignment_loss‚0.14771807495614658 1loss/SRqqt2/detection_loss‚0.7424591667478444 /loss/FBt2/detection_loss‚1.1918660423438128 trainer/global_step‚649 % _timestamp‚1.7843580631732423e+09 0loss/FRt2/assignment_loss‚2.3068227222012707 /loss/FRt2/detection_loss‚1.5240089412301083 /loss/SRqqt1/detection_loss‚1.63471594485319 &loss/total_loss‚16.694386973291877 epoch‚1 _step‚524Vë’¨‚(:‡ ½ÉìÒ‚ñ¼ gpu.0.gpu‚90.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ1” ³©‚(:¨ ½Éìҧ* proc.memory.percent‚16.598868370056152 disk.drbd2165.in‚0 disk.dm-0.out‚ 657.892578125# disk.nvme0c0n1.in‚ 1174.03515625 cpu‚26.109036611122068 disk.dm-1.out‚ 12.71484375$ disk.nvme0c0n1.out‚ 657.892578125 disk.sda2.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 9.83984375 network.sent‚925139* disk./.usagePercent‚39.334213084450866! disk.nvme0n1.in‚ 1174.03515625 disk.sda3.out‚ 12.71484375 proc.cpu.threads‚17 disk.dm-0.in‚ 1174.03515625 disk.sda.out‚ 12.71484375$ memory_percent‚99.93994235992432 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2719.55859375 disk.rbd1.out‚1.25 disk.sda2.in‚0 network.recv‚200288$ disk./.usageGB‚586.1082000732422 disk.dm-1.in‚0 disk.rbd1.in‚0" disk.nvme0n1.out‚ 657.892578125 disk.sda.in‚0 disk.sda3.in‚0½;“Ÿª‚(:” ÌÉìÒÁý›» gpu.0.gpu‚41.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N2À9®«‚(:£ ÌÉìÒºŽå¼ disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 663.169921875 cpu‚26.106800735739416 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.33420566023237 disk.sda3.out‚ 13.12890625$ disk./.usageGB‚586.1080894470215 disk.sda.out‚ 13.12890625 disk.dm-0.out‚ 663.169921875" disk.nvme0n1.out‚ 663.169921875 network.sent‚977722 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1174.03515625 network.recv‚205301 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0$ memory_percent‚99.94266033172607 disk.sda1.in‚0 disk.dm-1.out‚ 13.12890625 disk.rbd1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.598892211914062 disk.dm-0.in‚ 1174.03515625 disk.rbd1.out‚1.25# disk.nvme0c0n1.in‚ 1174.03515625& proc.memory.availableMB‚ 9.39453125 proc.memory.rssMB‚ 2719.5625äêD+0¬+ _wandbruntime‚595  _runtime‚595žæÙÎÅ­¿ trainer/global_step‚699 $ _timestamp‚1.784358103351737e+09 _runtime‚ 595.427089585 _step‚53 ' lr-AdamW/pg1‚0.00043475004305978357 ' lr-AdamW/pg2‚0.00043475004305978357ý ©É®Ã ' lr-AdamW/pg1‚0.00043475004305978357 ' lr-AdamW/pg2‚0.00043475004305978357 trainer/global_step‚699 $ _timestamp‚1.784358103351737e+09 _runtime‚ 595.427089585 _step‚535K÷Ï0¯+ _wandbruntime‚595  _runtime‚595ú•æÆ'°ÀO YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24276801312256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104501417340928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6429437 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105228699073351 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23540003334144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6439412 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23540021946368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105034128375808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105228341484109 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104501400301568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 $ _timestamp‚1.784358103352508e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105762530923008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2723077 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117676032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105762530923008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2722813 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 736797978112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6439412 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1493176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1229898 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728061209088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726941182541 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105034111336448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6429437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105034128375808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105034111336448 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728402547200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105228341484109 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104501400301568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 $ _timestamp‚1.784358103352508e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105762530923008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2723077 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117676032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105762530923008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2722813 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 736797978112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6439412 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1493176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1229898 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728061209088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726941182541 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105034111336448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6429437 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚699 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105034128375808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105034111336448 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728402547200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 596.230504274 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1494941 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843581041549315e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105181352366080 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2726857 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 _step‚55 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105910784586752 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105375863209703 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1495211 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24311252244480 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728292786257 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5474560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105181369405440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 737820151808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104647928012800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24311361524736 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105181352366080 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5474560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 ZMDeviceStatsMonitor.on_train_batch_enaÄN•.d/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90668032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647910973440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729415181312 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5474558 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24311361524736 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105181352366080 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5474560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90668032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647910973440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729415181312 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5474558 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 596.230504274 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1494941 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843581041549315e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105181352366080 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2726857 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 _step‚55 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105910784586752 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105375863209703 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1495211 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24311252244480 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728292786257 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5474560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105181369405440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 737820151808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104647928012800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚07-Èm¤0µ+ _wandbruntime‚596  _runtime‚596þÞ¡¶ú 1loss/SRqqt1/assignment_loss‚1.292858582271806 1loss/FBt2/assignment_loss‚0.13645678622628557 /loss/FRt2/detection_loss‚1.5478522991308112 1loss/SRqqt2/detection_loss‚0.7955456329657837 epoch‚1 _step‚56 2loss/SRqqt2/assignment_loss‚0.9403392557812539 /loss/FBt2/detection_loss‚2.0818913836260493 .loss/FRt1/assignment_loss‚3.16444607851977 1loss/SRqqt1/detection_loss‚1.4333307249849574 /loss/FBt1/detection_loss‚4.0272367402911184 /loss/FRt2/assignment_loss‚2.031492392891802 0loss/FBt1/assignment_loss‚0.5303570431104163 /loss/FRt1/detection_loss‚0.6702481898108129 &loss/total_loss‚18.652055109610863 trainer/global_step‚699 % _timestamp‚1.7843581041582391e+09 _runtime‚ 596.2332830714#O·þ 0loss/FBt1/assignment_loss‚0.5303570431104163 /loss/FRt1/detection_loss‚0.6702481898108129 &loss/total_loss‚18.652055109610863 trainer/global_step‚699 % _timestamp‚1.7843581041582391e+09 _runtime‚ 596.233283071 1loss/SRqqt1/assignment_loss‚1.292858582271806 1loss/FBt2/assignment_loss‚0.13645678622628557 /loss/FRt2/detection_loss‚1.5478522991308112 1loss/SRqqt2/detection_loss‚0.7955456329657837 epoch‚1 _step‚56 2loss/SRqqt2/assignment_loss‚0.9403392557812539 /loss/FBt2/detection_loss‚2.0818913836260493 .loss/FRt1/assignment_loss‚3.16444607851977 1loss/SRqqt1/detection_loss‚1.4333307249849574 /loss/FBt1/detection_loss‚4.0272367402911184 /loss/FRt2/assignment_loss‚2.03149239289180285¡Y«¸‚(:  ÛÉìÒ¬ýʼ# disk.nvme0n1.out‚666.2900390625 disk.sda3.out‚ 13.421875 disk.sda1.out‚0# memory_percent‚99.9427318572998 proc.memory.rssMB‚ 2719.5625* proc.memory.percent‚16.598892211914062* disk./.usagePercent‚39.334203100157026 disk.sda.out‚ 13.421875 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 13.421875# disk.nvme0c0n1.in‚ 1174.03515625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚983355$ disk./.usageGB‚586.1080513000488 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1174.03515625 disk.dm-0.in‚ 1174.03515625% proc.memory.availableMB‚ 9.3828125 disk.sda2.in‚0 cpu‚26.10145040291918 disk.rbd1.out‚1.3125 disk.drbd2165.out‚0 disk.dm-1.in‚0 network.recv‚210064 disk.rbd0.in‚0% disk.nvme0c0n1.out‚666.2900390625 disk.dm-0.out‚666.2900390625 disk.rbd1.in‚0 disk.sda3.in‚0G©”¹‚(:‰ ÛÉìÒ»‚ç¼ gpu.0.gpu‚67.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y`¾“º‚(:ˆ êÉìÒ¹¢³º gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yžt¶»»‚(:° êÉìÒ¾…¼& disk.nvme0c0n1.out‚671.17138671875 disk.sda3.out‚ 13.93359375 network.sent‚ 1.046484e+06! disk.nvme0n1.in‚ 1313.18359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚671.17138671875$ memory_percent‚99.94268417358398 disk.sda.out‚ 13.93359375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚1.3125* proc.memory.percent‚16.598892211914062# disk.nvme0c0n1.in‚ 1313.18359375! disk.dm-0.out‚671.17138671875 disk.dm-1.out‚ 13.93359375 disk.rbd1.in‚0 disk.sda3.in‚0* disk./.usagePercent‚39.334211548405655 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1313.18359375$ proc.memory.availableMB‚9.390625$ disk./.usageGB‚586.1081771850586 disk.sda.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2719.5625 cpu‚26.101064110527588 network.recv‚215626êü^\“¼‚(:ˆ ùÉìÒÍš’» gpu.0.gpu‚16.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚245.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.1256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ÍÚ$´½‚(:© ùÉìÒ°šµ¼ disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 674.10546875# disk.nvme0c0n1.in‚ 1313.18359375* proc.memory.percent‚16.598892211914062 network.recv‚218920 disk.drbd2165.out‚0 disk.dm-1.out‚ 14.36328125$ memory_percent‚99.94018077850342 disk.drbd2165.in‚0 disk.rbd1.out‚1.4375 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 1.051499e+06 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 674.10546875! disk.nvme0n1.in‚ 1313.18359375 disk.sda.out‚ 14.36328125* disk./.usagePercent‚39.334209244337856$ disk./.usageGB‚586.1081428527832 disk.sda2.in‚0 disk.dm-0.in‚ 1313.18359375 disk.rbd1.in‚0 proc.memory.rssMB‚ 2719.5625 proc.cpu.threads‚17 disk.sda3.out‚ 14.36328125 cpu‚26.099487183316363! disk.nvme0n1.out‚ 674.10546875 disk.sda.in‚0& proc.memory.availableMB‚ 9.80078125&ön60¾+ _wandbruntime‚636  _runtime‚636ˊÿ½ & lr-AdamW/pg1‚0.0004346595604202478 & lr-AdamW/pg2‚0.0004346595604202478 trainer/global_step‚749 $ _timestamp‚1.784358144271747e+09 _runtime‚ 636.350219142 _step‚57L0w¯ÇÀÁ & lr-AdamW/pg1‚0.0004346595604202478 & lr-AdamW/pg2‚0.0004346595604202478 trainer/global_step‚749 $ _timestamp‚1.784358144271747e+09 _runtime‚ 636.350219142 _step‚579È¢é0Á+ _wandbruntime‚636  _runtime‚636xôšÅ'¿O D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 % _timestamp‚1.7843581442757535e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5848526 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2913220 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112396179857408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25211459872768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 636.35121892 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1596328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111826933895168 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6883889 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5848528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1317151 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 788088246272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111826950934528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5848526 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112396179857408 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112396162818048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779098049024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6893864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112604849153859 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚749 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6893864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113175277906432 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25999451414528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1317148 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚58 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779098049024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25211441260544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78092288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975|cîCÏÃÃO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112396179857408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25211459872768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 636.35121892 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1596328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111826933895168 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6883889 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/¶}‰úallocation.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5848528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1317151 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 788088246272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111826950934528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5848526 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112396179857408 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112396162818048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779098049024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6893864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112604849153859 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚749 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6893864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113175277906432 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25999451414528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1317148 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚58 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779098049024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25211441260544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78092288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 % _timestamp‚1.7843581442757535e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5848526 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2913220 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0:-èNþ0Ä+ _wandbruntime‚636  _runtime‚636|+\ûÑ&ÅËM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1318896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12749099 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111973444567040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26033897808384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25244888631296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112752013290211 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6902930 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780110683136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 789116360192 % _timestamp‚1.7843581448897512e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112543403847680 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 _runtime‚ 636.965299387 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 789027789312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112543420887040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113323531570176 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5856160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112543403847680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113323531570176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6892941 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26034004991488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474?wþ%Õ&ÆÏM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1318896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12749099 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111973444567040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26033897808384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25244888631296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112752013290211 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6902930 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780110683136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 789116360192 % _timestamp‚1.7843581448897512e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112543403847680 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 _runtime‚ 636.965299387 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 789027789312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112543420887040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113323531570176 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5856160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112543403847680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113323531570176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6892941 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26034004991488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736;‚õÅE0Ç+ _wandbruntime‚636  _runtime‚636™,Èû 1loss/FBt1/assignment_loss‚0.34557447648403467 1loss/FBt2/assignment_loss‚0.12693692391526318 /loss/FRt1/detection_loss‚0.6565427860348018 _runtime‚ 636.967623315 _step‚60 1loss/SRqqt1/assignment_loss‚1.105678144690051 1loss/SRqqt2/detection_loss‚0.7809438080784064 /loss/FBt2/detection_loss‚1.0089139086970198 &loss/total_loss‚14.823092434850444 trainer/global_step‚749 2loss/SRqqt2/assignment_loss‚0.7960450122685012 0loss/SRqqt1/detection_loss‚1.394322460925823 .loss/FBt1/detection_loss‚1.961876067519188 /loss/FRt1/assignment_loss‚3.290224861966971 0loss/FRt2/assignment_loss‚1.9273554542485405 /loss/FRt2/detection_loss‚1.4286785300218445 epoch‚1 % _timestamp‚1.7843581448925474e+09›‚>zÉÿ 2loss/SRqqt2/assignment_loss‚0.7960450122685012 0loss/SRqqt1/detection_loss‚1.394322460925823 .loss/FBt1/detection_loss‚1.961876067519188 /loss/FRt1/assignment_loss‚3.290224861966971 0loss/FRt2/assignment_loss‚1.9273554542485405 /loss/FRt2/detection_loss‚1.4286785300218445 epoch‚1 % _timestamp‚1.7843581448925474e+09 1loss/FBt1/assignment_loss‚0.34557447648403467 1loss/FBt2/assignment_loss‚0.12693692391526318 /loss/FRt1/detection_loss‚0.6565427860348018 _runtime‚ 636.967623315 _step‚60 1loss/SRqqt1/assignment_loss‚1.105678144690051 1loss/SRqqt2/detection_loss‚0.7809438080784064 /loss/FBt2/detection_loss‚1.0089139086970198 &loss/total_loss‚14.823092434850444 trainer/global_step‚749<'•T”Ê‚(:‰ ˆÊìҨð¼ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wøŽ-¹Ë‚(:® ˆÊìÒï‹ï¼ cpu‚26.09722080007616 disk.rbd1.in‚0$ disk.nvme0n1.out‚679.22509765625 disk.drbd2165.in‚0$ proc.memory.availableMB‚9.546875 proc.memory.rssMB‚ 2719.5625$ disk./.usageGB‚586.1074104309082 disk.rbd1.out‚1.4375 disk.sda.out‚ 14.70703125 disk.sda3.in‚0* proc.memory.percent‚16.598892211914062# disk.nvme0c0n1.in‚ 1313.18359375 disk.sda3.out‚ 14.70703125) disk./.usagePercent‚39.33416009089128 disk.rbd0.in‚0& disk.nvme0c0n1.out‚679.22509765625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1313.18359375! disk.dm-0.out‚679.22509765625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 14.70703125 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 1.056487e+06 network.recv‚222380 disk.sda2.out‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 1313.18359375$ memory_percent‚99.94173049926758$Þ¬›ŸÌ‚(:” —ÊìÒÌ«é» gpu.0.gpu‚37.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚199.034' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.806799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v·×–Í‚(:‹ —ÊìÒ–Ñ¢½ disk.dm-0.in‚ 1313.18359375% disk.nvme0c0n1.out‚681.9287109375 proc.memory.rssMB‚ 2719.5625 cpu‚26.08921312082408 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1313.18359375 disk.sda.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚1.5) disk./.usagePercent‚39.33418287556182 disk.drbd2165.out‚0 disk.dm-0.out‚681.9287109375 disk.rbd0.out‚0$ disk./.usageGB‚586.1077499389648 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 1.119345e+06# disk.nvme0c0n1.in‚ 1313.18359375* proc.memory.percent‚16.598892211914062 disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.availableMB‚9.84375 disk.sda.out‚15 disk.sda2.in‚0# memory_percent‚99.9399185180664# disk.nvme0n1.out‚681.9287109375 disk.sda3.out‚15 disk.dm-1.out‚15 network.recv‚227598 disk.sda1.in‚0ªœ£è$΂(:ˆ ¦ÊìÒÞô¿» gpu.0.gpu‚33.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smC‰¸ÚVolock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´cÁM§Ï‚(:œ ¦ÊìÒ«¾ñ¼ disk.sda2.out‚0$ memory_percent‚99.94137287139893* proc.memory.percent‚16.598892211914062 network.recv‚230839 disk.sda3.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 15.42578125) disk./.usagePercent‚39.33418287556182 disk.dm-0.in‚1463.75& disk.nvme0c0n1.out‚687.91845703125 disk.sda.in‚0 proc.memory.rssMB‚ 2719.5625 disk.rbd1.out‚1.5$ disk.nvme0n1.out‚687.91845703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0! disk.dm-0.out‚687.91845703125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚1463.75 proc.cpu.threads‚17 network.sent‚ 1.124305e+06$ disk./.usageGB‚586.1077499389648 disk.dm-1.out‚ 15.42578125 disk.sda3.out‚ 15.42578125& proc.memory.availableMB‚ 9.60546875 cpu‚26.081252178946574 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚1463.75°D‹0Ð+ _wandbruntime‚676  _runtime‚676nCÆÑÀ _runtime‚ 676.316839814 _step‚61 ' lr-AdamW/pg1‚0.00043455513002074964 ' lr-AdamW/pg2‚0.00043455513002074964 trainer/global_step‚799 % _timestamp‚1.7843581842415102e+09GØŠÊÒÄ trainer/global_step‚799 % _timestamp‚1.7843581842415102e+09 _runtime‚ 676.316839814 _step‚61 ' lr-AdamW/pg1‚0.00043455513002074964 ' lr-AdamW/pg2‚0.00043455513002074964=Yô00Ó+ _wandbruntime‚676  _runtime‚676ÔÁ¤Ç'ÔÁO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843581842427683e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 839249943040 trainer/global_step‚799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 828174156357 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119152484528128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6230128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 828514706239 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7348316 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107190272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119758214299648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1698836 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 _runtime‚ 676.318251778 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119758214299648 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119980999234367 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120588024889856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1699108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119758231339008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839338521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120588024889856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7348316 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120587666512384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268TìDË'ÕÅO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6230128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 828514706239 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7348316 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107190272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119758214299648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1698836 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 _runtime‚ 676.318251778 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119758214299648 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119980999234367 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120588024889856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1699108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119758231339008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839338521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120588024889856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7348316 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120587666512384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843581842427683e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 839249943040 trainer/global_step‚799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 828174156357 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119152484528128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0>¯y'0Ö+ _wandbruntime‚676  _runtime‚676¼DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120736278553600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1406143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119905472368640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 830806184960 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 830464839680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120128163370719 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1700864 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 829525760073 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119298978160640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6237758 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843581848555293e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _runtime‚ 676.931269656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26916326557696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ûq!HØÏM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120736278553600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 л#SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120736278553600 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1406143 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119905472368640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 830806184960 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 830464839680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120128163370719 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1700864 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 829525760073 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119298978160640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6237758 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843581848555293e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _runtime‚ 676.931269656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26916326557696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 Fì=ßæ{ =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9907326698303223 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11166666448116302 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.25868743658065796 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2620708644390106 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4814814329147339 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00917026586830616 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.32407405972480774 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017619285732507706 +jet/accuracy_2_of_3‚0.33013632893562317 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16652849316596985 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9811382293701172 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 particle/accuracy_1_of_1‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5555555820465088 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2447916716337204 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40764641761779785 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7719327807426453 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.41208332777023315 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07160855829715729 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.39889761805534363 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9224562048912048 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.981128990650177 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6896020174026489 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06762704998254776 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0807572677731514 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967347383499146 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15548856556415558 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1448584794998169 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6752861738204956 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9944546818733215 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12323515862226486 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5923585891723633 particle/accuracy_4_of_5‚1 *loss/val_total_loss‚15.399358862511484 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3931359648704529 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1854945570230484 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +particle/specificity‚0.8933913111686707 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4348733425140381 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.48655134439468384 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.38507962226867676 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.96122807264328 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9523809552192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07824943959712982 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.08287462592124939 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0621662475168705 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8564814329147339 jet/accuracy_3_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5796485543251038 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.328125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3914227783679962 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.42108961939811707 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.826269268989563 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚799 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8285917639732361 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07384729385375977 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3619471788406372 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9854647517204285 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.35885050892829895 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9843365550041199 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6663076281547546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08040989935398102 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06802399456501007 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.44247880578041077 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.35864192247390747 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20927736163139343 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9645423889160156 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5104166865348816 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4197908937931061 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42140793800354004 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.14436271786689758 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3848904073238373 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.34652024507522583 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3429605960845947 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18370676040649414 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17185378074645996 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4814814329147339 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21494454145431519 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03182627633213997 +jet/accuracy_2_of_2‚0.08622431010007858 /particle/accuracy_3_of_4‚0.9968253374099731 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3394922614097595 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.33630210161209106 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5400412678718567 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.40282541513442993 +particle/sensitivity‚0.5469940900802612 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48976656794548035 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022057289257645607 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4477226734161377 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5282312631607056 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4003647267818451 *jet/accuracy_3_of_4‚0.0474107563495636 ,jet/accuracy_4_of_4‚0.007130124606192112 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4523160457611084 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006888185162097216 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.40527352690696716 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 particle/accuracy_3_of_5‚1 _step‚65 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.43664678931236267 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9853174090385437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_2‚0.6473551988601685 *jet/accuracy_1_of_3‚0.8052773475646973 7validation_average_jet_accuracy‚0.23599116504192352 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9879751801490784 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.32362431287765503 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9908771514892578 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3696029484272003 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39989906549453735 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3857801556587219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.26712504029273987 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.26925787329673767 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014376540668308735 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003681079251691699 .particle/accuracy_3_of_3‚0.997450590133667 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3677018880844116 particle/accuracy_2_of_3‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.668820858001709 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9861059784889221 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7083333134651184 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.031248554587364197 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9821345210075378 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6764172315597534 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9936507940292358 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +jet/accuracy_1_of_1‚0.18178574740886688 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.38518592715263367 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3288268446922302 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8682928681373596 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5647449493408203 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_3‚0.0465645007789135 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9491835236549377 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36260151863098145 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06164001300930977 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6671147346496582 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6111111044883728 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2953685224056244 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.29284971952438354 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3729640543460846 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.40419715642929077 *jet/accuracy_1_of_4‚0.8511640429496765 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.35864192247390747 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843582021323166e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9507473707199097 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5555555820465088 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2831454277038574 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2831454277038574 *jet/accuracy_2_of_4‚0.5758670568466187 /particle/accuracy_2_of_4‚0.9968253374099731 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.37577593326568604 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.967380702495575 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9689759016036987 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8469300866127014 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9944546818733215 particle/accuracy_1_of_3‚1 'particle/f_score‚0.5648305416107178 _runtime‚ 694.208276621 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05257600173354149 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3614600896835327 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 /particle/accuracy_4_of_4‚0.8914777040481567 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0653940737247467 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06802399456501007 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34490740299224854 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6193625330924988 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9799525737762451 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9775116443634033 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4345531463623047 (particle/accuracy‚0.8190026879310608 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8564814329147339 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9947806596755981 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3332292139530182 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15189692378044128 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14144957065582275 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6727702021598816 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.44247880578041077 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9928145408630371U»ê7xàê{ 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9775116443634033 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4345531463623047 (particle/accuracy‚0.8190026879310608 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8564814329147339 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9947806596755981 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3332292139530182 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15189692378044128 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14144957065582275 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6727702021598816 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.44247880578041077 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9928145408630371 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9907326698303223 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11166666448116302 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.25868743658065796 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2620708644390106 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4814814329147339 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00917026586830616 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.32407405972480774 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017619285732507706 +jet/accuracy_2_of_3‚0.33013632893562317 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16652849316596985 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9811382293701172 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 particle/accuracy_1_of_1‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5555555820465088 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2447916716337204 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40764641761779785 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7719327807426453 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.41208332777023315 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07160855829715729 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.39889761805534363 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9224562048912048 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.981128990650177 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6896020174026489 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06762704998254776 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0807572677731514 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967347383499146 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15548856556415558 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1448584794998169 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6752861738204956 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9944546818733215 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12323515862226486 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5923585891723633 particle/accuracy_4_of_5‚1 *loss/val_total_loss‚15.399358862511484 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3931359648704529 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/2FRt2SRqqt1FBt/eveK»Ñx+nt_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1854945570230484 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +particle/specificity‚0.8933913111686707 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4348733425140381 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.48655134439468384 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.38507962226867676 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.96122807264328 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9523809552192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07824943959712982 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.08287462592124939 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0621662475168705 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8564814329147339 jet/accuracy_3_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5796485543251038 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.328125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3914227783679962 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.42108961939811707 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.826269268989563 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚799 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8285917639732361 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07384729385375977 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3619471788406372 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9854647517204285 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.35885050892829895 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9843365550041199 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6663076281547546 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08040989935398102 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06802399456501007 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.44247880578041077 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.35864192247390747 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20927736163139343 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9645423889160156 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5104166865348816 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4197908937931061 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42140793800354004 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.14436271786689758 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3848904073238373 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.34652024507522583 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3429605960845947 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18370676040649414 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17185378074645996 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4814814329147339 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21494454145431519 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03182627633213997 +jet/accuracy_2_of_2‚0.08622431010007858 /particle/accuracy_3_of_4‚0.9968253374099731 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3394922614097595 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.33630210161209106 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5400412678718567 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.40282541513442993 +particle/sensitivity‚0.5469940900802612 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48976656794548035 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022057289257645607 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4477226734161377 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5282312631607056 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4003647267818451 *jet/accuracy_3_of_4‚0.0474107563495636 ,jet/accuracy_4_of_4‚0.007130124606192112 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4523160457611084 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006888185162097216 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.40527352690696716 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 particle/accuracy_3_of_5‚1 _step‚65 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.43664678931236267 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9853174090385437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_2‚0.6473551988601685 *jet/accuracy_1_of_3‚0.8052773475646973 7validation_average_jet_accuracy‚0.23599116504192352 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9879751801490784 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.32362431287765503 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9908771514892578 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3696029484272003 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.39989906549453735 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3857801556587219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.26712504029273987 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.26925787329673767 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014376540668308735 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003681079251691699 .particle/accuracy_3_of_3‚0.997450590133667 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3677018880844116 particle/accuracy_2_of_3‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.668820858001709 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9861059784889221 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7083333134651184 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.031248554587364197 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9821345210075378 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6764172315597534 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9936507940292358 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +jet/accuracy_1_of_1‚0.18178574740886688 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.38518592715263367 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3288268446922302 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8682928681373596 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5647449493408203 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_3‚0.0465645007789135 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9491835236549377 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36260151863098145 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06164001300930977 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6671147346496582 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6111111044883728 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2953685224056244 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.29284971952438354 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3729640543460846 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.40419715642929077 *jet/accuracy_1_of_4‚0.8511640429496765 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.35864192247390747 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7843582021323166e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9507473707199097 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5555555820465088 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2831454277038574 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2831454277038574 *jet/accuracy_2_of_4‚0.5758670568466187 /particle/accuracy_2_of_4‚0.9968253374099731 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.37577593326568604 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.967380702495575 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9689759016036987 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8469300866127014 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9944546818733215 particle/accuracy_1_of_3‚1 'particle/f_score‚0.5648305416107178 _runtime‚ 694.208276621 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05257600173354149 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3614600896835327 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 /particle/accuracy_4_of_4‚0.8914777040481567 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0653940737247467 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06802399456501007 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34490740299224854 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6193625330924988 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9799525737762451A»½ß”á‚(:‰ ÄÊìÒÞÑÒ¼ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_?&[«â‚(:  ÄÊìÒ꯽ disk.dm-0.in‚1463.75 network.recv‚238606* disk./.usagePercent‚39.350318774437454$ disk.nvme0n1.out‚1190.8994140625! disk.dm-0.out‚1190.8994140625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚96.97637557983398& proc.memory.availableMB‚ 495.390625 network.sent‚ 1.191925e+06 disk.sda3.out‚ 16.8984375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 cpu‚26.156368474495405 disk.nvme0n1.in‚1463.75 disk.sda2.out‚0 disk.dm-1.out‚ 16.8984375 disk.rbd1.out‚1.5625) proc.memory.percent‚16.57559871673584 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚586.3481864929199 disk.nvme0c0n1.in‚1463.75& disk.nvme0c0n1.out‚1190.8994140625 disk.sda.out‚ 16.8984375# proc.memory.rssMB‚ 2715.74609375E\Šü”ã‚(:‰ ÓÊìÒ߯˻ gpu.0.gpu‚57.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ßÇ|¨ä‚(: ÓÊìÒé£î» proc.cpu.threads‚17 disk.sda.out‚ 17.96484375 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚1.625( proc.memory.availableMB‚ 495.19140625 cpu‚26.1476185903985% disk.nvme0c0n1.out‚1195.111328125$ disk./.usageGB‚586.3481903076172 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚39.35031903044499 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 17.96484375# memory_percent‚96.9775915145874 network.sent‚ 1.249214e+06 disk.nvme0n1.in‚1463.75 network.recv‚243099 disk.sda3.out‚ 17.96484375 disk.sda2.out‚0 disk.dm-0.out‚1195.111328125# proc.memory.rssMB‚ 2715.74609375) proc.memory.percent‚16.57559871673584 disk.rbd1.in‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚1195.111328125 disk.dm-0.in‚1463.75 disk.nvme0c0n1.in‚1463.75£—å‚(:’ âÊìÒ¼¿¼ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.876' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.975199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rv“0Áæ‚(:¶ âÊìÒéÑʽ disk.sda3.out‚ 19.07421875 disk.sda.in‚0& proc.memory.availableMB‚ 495.328125 disk.rbd1.out‚1.625 disk.drbd2165.out‚0 disk.rbd0.in‚0 network.recv‚246570 disk.dm-1.in‚0 disk.dm-1.out‚ 19.07421875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1587.01171875% disk.nvme0n1.out‚1201.65478515625$ memory_percent‚96.97675704956055 disk.rbd0.out‚0) proc.memory.percent‚16.57559871673584 disk.sda.out‚ 19.07421875 disk.sda1.out‚0 network.sent‚ 1.254228e+06) disk./.usagePercent‚39.35033874302513$ disk./.usageGB‚586.3484840393066 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1587.01171875 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1587.01171875" disk.dm-0.out‚1201.65478515625 disk.sda3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2715.74609375 cpu‚26.139311263244096 proc.cpu.threads‚17' disk.nvme0c0n1.out‚1201.65478515625™É£0ç+ _wandbruntime‚735  _runtime‚735÷[GÞÆèÀ ' lr-AdamW/pg2‚0.00043443675856865837 trainer/global_step‚849 % _timestamp‚1.7843582434003024e+09 _runtime‚ 735.477146156 _step‚66 ' lr-AdamW/pg1‚0.00043443675856865837ˆ"'/ÊéÄ ' lr-AdamW/pg1‚0.00043443675856865837 ' lr-AdamW/pg2‚0.00043443675856865837 trainer/global_step‚849 % _timestamp‚1.7843582434003024e+09 _runtime‚ 735.477146156 _step‚66BîI0ê+ _wandbruntime‚735  _runtime‚735²ûºyÆ'ëÀO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1494168 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129049849470976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28570247825408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 898222258688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897880920064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129049832431616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1835518 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 trainer/global_step‚849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7873091 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897880920064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129947713351680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843582434010928e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 908666919936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 908588828160 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚15087Á žìÄO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6732392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1494171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6732394 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28570247825408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 898222258688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897880920064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129049832431616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1835518 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 trainer/global_step‚849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7873091 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897880920064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129947713351680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843582434010928e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 908666919936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 908588828160 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1494168 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129049849470976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898893547008 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3333500 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128533174968320 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7872167 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129431053979891 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 897861972211 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6740026 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 % _timestamp‚1.7843582440132103e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898893547008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚68 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6740026 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129197090500608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6740024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1837315 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28603657971712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14612191 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚130096325393408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6740024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1495913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129197073461248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130095967008256 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7872167$c£}Õ&ïÏM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 % _timestamp‚1.7843582440132103e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898893547008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚68 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6740026 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129197090500608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6740024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1837315 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28603657971712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14612191 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚130096325393408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6740024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1495913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129197073461248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130095967008256 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7872167 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29513262374912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28603676583936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128533192007680 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7882157 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14612191 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899234892800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 736.088761132 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 trainer/global_step‚849 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7882157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1495916 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129430696390533 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898893547008 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3333500 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128533174968320 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7872167 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129431053979891 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 897861972211 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6740026 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437DnÑ­}0ð+ _wandbruntime‚736  _runtime‚736—«ññý 2loss/SRqqt1/assignment_loss‚1.0733538615694713 2loss/SRqqt2/assignment_loss‚0.7976768396657156 epoch‚2 % _timestamp‚1.7843582440155685e+09 1loss/FBt1/assignment_loss‚0.24186456064884018 1loss/FBt2/assignment_loss‚0.12885159011911423 .loss/FBt1/detection_loss‚1.872429482638836 /loss/FBt2/detection_loss‚0.6259366169778429 trainer/global_step‚849 _step‚69 0loss/FRt1/assignment_loss‚3.0923366443635096 /loss/FRt2/assignment_loss‚1.676898883039301 /loss/FRt2/detection_loss‚1.3606144299633294 1loss/SRqqt1/detection_loss‚1.3351531089061783 1loss/SRqqt2/detection_loss‚0.8576218994689375 &loss/total_loss‚13.768683562819811 _runtime‚ 736.090628288 /loss/FRt1/detection_loss‚0.7059456454587367–?ò 2loss/SRqqt2/assignment_loss‚0.7976768396657156 epoch‚2 % _timestamp‚1.7843582440155685e+09 1loss/FBt1/assignment_loss‚0.24186456064884018 1loss/FBt2/assignment_loss‚0.12885159011911423 .loss/FBt1/detection_loss‚1.872429482638836 /loss/FBt2/detection_loss‚0.6259366169778429 trainer/global_step‚849 _step‚69 0loss/FRt1/assignment_loss‚3.0923366443635096 /loss/FRt2/assignment_loss‚1.676898883039301 /loss/FRt2/detection_loss‚1.3606144299633294 1loss/SRqqt1/detection_loss‚1.3351531089061783 1loss/SRqqt2/detection_loss‚0.8576218994689375 &loss/total_loss‚13.768683562819811 _runtime‚ 736.090628288 /loss/FRt1/detection_loss‚0.7059456454587367 2loss/SRqqt1/assignment_loss‚1.0733538615694713Eö‚§”ó‚(:‰ ñÊìÒã궺 gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=n~p¿ô‚(:´ ñÊìÒÅåä¼) proc.memory.percent‚16.57559871673584$ memory_percent‚98.48310947418213 network.recv‚249778% disk.nvme0n1.out‚1206.91455078125 disk.sda1.in‚0 disk.sda2.out‚0" disk.dm-0.out‚1206.91455078125$ disk./.usageGB‚586.3483581542969 disk.rbd1.in‚0' disk.nvme0c0n1.out‚1206.91455078125! disk.nvme0n1.in‚ 1587.01171875( proc.memory.availableMB‚ 248.52734375 disk.rbd0.out‚0 disk.dm-1.in‚0 cpu‚26.13118049117282 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-0.in‚ 1587.01171875 network.sent‚ 1.259023e+06( disk./.usagePercent‚39.3503302947765# disk.nvme0c0n1.in‚ 1587.01171875 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 19.3984375 disk.sda3.out‚ 19.3984375 disk.sda1.out‚0 disk.rbd1.out‚1.6875 disk.sda2.in‚0 disk.dm-1.out‚ 19.3984375# proc.memory.rssMB‚ 2715.74609375`U_žõ‚(:“ €ËìÒüÚã» gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.509999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä• :»ö‚(:° €ËìÒ±Ù¥¾& proc.memory.availableMB‚ 248.671875 disk.sda2.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚1214.3994140625 disk.sda1.out‚0! disk.dm-0.out‚1214.3994140625 disk.dm-1.out‚ 19.6953125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 1587.01171875) proc.memory.percent‚16.57559871673584 network.recv‚253997) disk./.usagePercent‚39.35031058219636$ disk./.usageGB‚586.3480644226074 disk.drbd2165.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 19.6953125$ memory_percent‚98.48222732543945# proc.memory.rssMB‚ 2715.74609375 cpu‚26.123514832982075 disk.dm-0.in‚ 1587.01171875 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚1214.3994140625 network.sent‚ 1.32205e+06 disk.sda1.in‚0 disk.rbd1.out‚1.6875! disk.nvme0n1.in‚ 1587.01171875 disk.sda3.out‚ 19.6953125Zm¨0÷+ _wandbruntime‚775  _runtime‚775”(íÆøÀ _step‚70 ' lr-AdamW/pg1‚0.00043430445366675135 ' lr-AdamW/pg2‚0.00043430445366675135 trainer/global_step‚899 % _timestamp‚1.7843582833134327e+09 _runtime‚ 775.388803991y/ ýÊùÄ ' lr-AdamW/pg1‚0.00043430445366675135 ' lr-AdamW/pg2‚0.00043430445366675135 trainer/global_step‚899 % _timestamp‚1.7843582833134327e+09 _runtime‚ 775.388803991 _step‚70F@i«B0ú+ _wandbruntime‚775  _runtime‚7754Š„߸ûÀO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8317567 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31201712508416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1939308 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1581418 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1939563 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137360460335104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 775.393128951 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30241667139584 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959946567680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137360818713088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8317567 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 947126855399 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135712214929408 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 947467405285 % _timestamp‚1.7843582833146825e+09‘¹žêÊ'üÄO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137360460335104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 775.393128951 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30241667139584 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959946567680 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137360818713088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8317567 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 947126855399 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135712214929408 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 947467405285 % _timestamp‚1.7843582833146825e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8317567 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31201712508416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1939308 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1581418 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1939563 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268G)ˆê½0ý+ _wandbruntime‚776  _runtime‚776ÚðôÑ&þËM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3524760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135858725601280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8336609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137508713991680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚136559141982208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135858708561920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8326619 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136559124942848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136559124942848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 _step‚72 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136807204060399 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961056742400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31236171252736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137509072376832 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121624 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8336609 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843582839274163e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1941594 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278912 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961056742400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31236171252736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137509072376832 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121624 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8336609 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843582839274163e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1941594 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278912 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3524760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135858725601280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8336609 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137508713991680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚136559141982208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135858708561920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8326619 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136559124942848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136559124942848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100HË;Ï0€+ _wandbruntime‚776  _runtime‚776qa±Xü 2loss/SRqqt2/assignment_loss‚0.7358169423098119 1loss/FBt1/assignment_loss‚0.23212759874762992 /loss/FRt2/detection_loss‚1.5207169800470888 1loss/SRqqt2/detection_loss‚0.8081957087146513 1loss/SRqqt1/detection_loss‚1.3776202526081491 _step‚73 /loss/FBt1/detection_loss‚1.6599848596034226 /loss/FBt2/detection_loss‚0.8909994603582118 epoch‚2 1loss/SRqqt1/assignment_loss‚0.980206474032322 /loss/FRt1/detection_loss‚0.6450053232579491 % _timestamp‚1.7843582839295976e+09 _runtime‚ 776.004679354 1loss/FBt2/assignment_loss‚0.17161903499484765 %loss/total_loss‚13.76997985465454 trainer/global_step‚899 /loss/FRt1/assignment_loss‚3.126109528132701 0loss/FRt2/assignment_loss‚1.6215776918477534àßt‰‚€ /loss/FRt1/assignment_loss‚3.126109528132701 0loss/FRt2/assignment_loss‚1.6215776918477534 2loss/SRqqt2/assignment_loss‚0.7358169423098119 1loss/FBt1/assignment_loss‚0.23212759874762992 /loss/FRt2/detection_loss‚1.5207169800470888 1loss/SRqqt2/detection_loss‚0.8081957087146513 1loss/SRqqt1/detection_loss‚1.3776202526081491 _step‚73 /loss/FBt1/detection_loss‚1.6599848596034226 /loss/FBt2/detection_loss‚0.8909994603582118 epoch‚2 1loss/SRqqt1/assignment_loss‚0.980206474032322 /loss/FRt1/detection_loss‚0.6450053232579491 % _timestamp‚1.7843582839295976e+09 _runtime‚ 776.004679354 1loss/FBt2/assignment_loss‚0.17161903499484765 %loss/total_loss‚13.76997985465454 trainer/global_step‚899IÐt?È”ƒ‚(:‰ ËìҌ藺 gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÀÓ¶„‚(:« ËìÒ›°§½ disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 cpu‚26.11506340320591 proc.cpu.threads‚17 disk.sda1.in‚0' proc.memory.availableMB‚ 248.4140625 disk.rbd0.out‚0 disk.sda.out‚ 19.921875 network.sent‚ 1.326977e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-0.out‚1218.431640625$ memory_percent‚98.48380088806152) disk./.usagePercent‚39.38662909104136$ disk./.usageGB‚586.8892364501953 disk.rbd0.in‚0 disk.sda3.out‚ 19.921875 disk.sda1.out‚0 disk.dm-0.in‚ 1587.01171875 disk.dm-1.out‚ 19.921875# proc.memory.rssMB‚ 2715.74609375 network.recv‚257427 disk.rbd1.out‚1.8125% disk.nvme0c0n1.out‚1218.431640625! disk.nvme0n1.in‚ 1587.01171875# disk.nvme0n1.out‚1218.431640625 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.57559871673584# disk.nvme0c0n1.in‚ 1587.01171875Ro;@”…‚(:‰ žËìÒ•Ó¹» gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00c–š¯†‚(:¤ žËìÒáÃÿ¼ disk.rbd0.out‚0$ disk./.usageGB‚587.4871597290039 disk.nvme0n1.in‚ 1773.8125 disk.sda3.out‚ 20.296875 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚98.48384857177734 proc.cpu.threads‚17 network.recv‚261719 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1773.8125 disk.drbd2165.in‚0% proc.memory.availableMB‚ 248.40625% disk.nvme0n1.out‚2391.16064453125 disk.sda.out‚ 20.296875 disk.sda2.out‚0# proc.memory.rssMB‚ 2715.74609375) proc.memory.percent‚16.57559871673584' disk.nvme0c0n1.out‚2391.16064453125 disk.sda.in‚0 cpu‚26.107790453894665) disk./.usagePercent‚39.42675622397325 disk.rbd1.out‚1.8125 disk.nvme0c0n1.in‚ 1773.8125 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚2391.16064453125 network.sent‚ 1.391857e+06 disk.dm-1.out‚ 20.296875Øê;Ÿ‡‚(:” ­ËìÒ­÷È» gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qsä󧈂(:œ ­ËìÒ…›½$ disk./.usageGB‚587.4478492736816 disk.sda2.out‚0 disk.dm-0.out‚ 2394.13671875 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 2394.13671875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 1773.8125" disk.nvme0n1.out‚ 2394.13671875 disk.sda.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 248.16015625( disk./.usagePercent‚39.4241180663329 disk.sda3.out‚ 20.890625 disk.dm-0.in‚ 1773.8125# proc.memory.rssMB‚ 2715.74609375 network.recv‚265251 disk.rbd0.out‚0 cpu‚26.101044649156165 network.sent‚ 1.394821e+06 disk.rbd1.out‚1.875) proc.memory.percent‚16.57559871673584 disk.sda.out‚ 20.890625 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 20.890625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 1773.8125$ memory_percent‚98.48535060882568WB!ô0‰+ _wandbruntime‚815  _runtime‚815 eÑÆŠÀ _step‚74 ' lr-AdamW/pg1‚0.00043415822381272505 ' lr-AdamW/pg2‚0.00043415822381272505 trainer/global_step‚949 % _timestamp‚1.7843583230084786e+09 _runtime‚ 815.083820554/ÛÔjÊ‹Ä ' lr-AdamW/pg1‚0.00043415822381272505 ' lr-AdamW/pg2‚0.00043415822381272505 trainer/global_step‚949 % _timestamp‚1.7843583230084786e+09 _runtime‚ 815.083820554 _step‚74JçbÍã0Œ+ _wandbruntime‚815  _runtime‚815ásÔÈ'ÂO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143773952434176 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 999271923712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71800320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495594 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8781995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143773935394816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31913123678208 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7495594 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8781995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8772019 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144773565696512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143037748523008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997743342307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998083892193 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 _runtime‚ 815.085180977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32924541836800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31913105065984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3711810 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 999613262336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1668671 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144035832415201 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143037731483648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143773952434176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144773207318528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1668668 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1011436770816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8772019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 % _timestamp‚1.7843583230096726e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚949 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 999613262336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 _step‚75 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8781995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143773935394816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31913123678208 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7495594 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8781995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8772019 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144773565696512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143037748523008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997743342307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998083892193 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 _runtime‚ 815.085180977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32924541836800 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31913105065984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3711810 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 999613262336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1668671 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144035832415201 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143037731483648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143773952434176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144773207318528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1668668 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1011436770816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8772019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 % _timestamp‚1.7843583230096726e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0KHFX0+ _wandbruntime‚815  _runtime‚815˜Ó°cò ÑM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3715853 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143921193463808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1670416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144183354140907 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7503226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8781071 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144921819360256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16284295 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1000625896448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2045437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143184259194880 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 998754396029 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7503224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 _step‚76 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31946552436736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 _runtime‚ 815.697834183 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86473216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143184242155520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7503224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144182996551549 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843583236223125e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143921176424448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1000625896448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1012543371776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105085440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8791061 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1012456898560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86473216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143184242155520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7503224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144182996551549 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843583236223125e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143921176424448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1000625896448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1012543371776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105085440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8791061 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1012456898560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3715853 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143921193463808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1670416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144183354140907 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7503226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0LG¡¥“0’+ _wandbruntime‚815  _runtime‚815!€/“û 2loss/SRqqt2/assignment_loss‚0.7247067014184198 /loss/FRt1/detection_loss‚0.6744568233529756 .loss/FBt1/detection_loss‚1.486885829464249 0loss/FRt1/assignment_loss‚3.1304463452898186 2loss/SRqqt1/assignment_loss‚0.8966756138920479 .loss/FRt2/detection_loss‚1.382631737460791 1loss/SRqqt2/detection_loss‚0.6945035995089823 trainer/global_step‚949 0loss/FRt2/assignment_loss‚1.6198633239209224 0loss/FBt2/assignment_loss‚0.0644738394637885 &loss/total_loss‚12.773541054017786 epoch‚2 $ _timestamp‚1.784358323624134e+09 _step‚77 1loss/FBt1/assignment_loss‚0.23190650991460995 1loss/SRqqt1/detection_loss‚1.2422095466790528 /loss/FBt2/detection_loss‚0.6247811836521284 _runtime‚ 815.699191961:j_”ÿ /loss/FRt1/detection_loss‚0.6744568233529756 .loss/FBt1/detection_loss‚1.486885829464249 0loss/FRt1/assignment_loss‚3.1304463452898186 2loss/SRqqt1/assignment_loss‚0.8966756138920479 .loss/FRt2/detection_loss‚1.382631737460791 1loss/SRqqt2/detection_loss‚0.6945035995089823 trainer/global_step‚949 0loss/FRt2/assignment_loss‚1.6198633239209224 0loss/FBt2/assignment_loss‚0.0644738394637885 &loss/total_loss‚12.773541054017786 epoch‚2 $ _timestamp‚1.784358323624134e+09 _step‚77 1loss/FBt1/assignment_loss‚0.23190650991460995 1loss/SRqqt1/detection_loss‚1.2422095466790528 /loss/FBt2/detection_loss‚0.6247811836521284 _runtime‚ 815.699191961 2loss/SRqqt2/assignment_loss‚0.7247067014184198M\¾õ“•‚(:ˆ ¼ËìÒ³²¨» gpu.0.gpu‚32.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝKÔ7ª–‚(:Ÿ ¼ËìÒ˧©¼ disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.out‚2398.537109375 proc.cpu.threads‚17$ disk./.usageGB‚586.4254760742188# disk.nvme0n1.out‚2398.537109375# proc.memory.rssMB‚ 2715.74609375 network.recv‚268445 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 21.171875 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda1.in‚0( disk./.usagePercent‚39.3555057430887 disk.rbd1.out‚1.875% disk.nvme0c0n1.out‚2398.537109375 network.sent‚ 1.399728e+06 disk.rbd0.out‚0 disk.dm-1.in‚0) proc.memory.percent‚16.57559871673584 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 21.171875 disk.sda2.in‚0$ memory_percent‚98.48415851593018( proc.memory.availableMB‚ 248.35546875 disk.nvme0n1.in‚ 1773.8125 disk.sda.out‚ 21.171875 disk.dm-0.in‚ 1773.8125 cpu‚26.094019174479822 disk.nvme0c0n1.in‚ 1773.8125ǸWʵ—‚(:ª ËËìÒ±«¯¼$ disk./.usageGB‚586.3870697021484 disk.dm-1.out‚ 21.44921875 disk.rbd0.in‚0) disk./.usagePercent‚39.35292825923396! disk.dm-0.out‚2402.3603515625 disk.nvme0c0n1.in‚ 1773.8125$ disk.nvme0n1.out‚2402.3603515625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1773.8125 disk.sda3.in‚0 disk.sda3.out‚ 21.44921875( proc.memory.availableMB‚ 248.37890625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚2402.3603515625 disk.nvme0n1.in‚ 1773.8125 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.percent‚16.57559871673584 network.recv‚273049 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 21.44921875 disk.drbd2165.out‚0 disk.sda2.in‚0 cpu‚26.087756423285438 network.sent‚ 1.462506e+06 disk.drbd2165.in‚0 disk.rbd1.out‚1.9375 disk.sda.in‚0$ memory_percent‚98.48401546478271# proc.memory.rssMB‚ 2715.74609375p–ì(˜‚(:’ ËËìÒ¢Âç¼ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñu`_0™+ _wandbruntime‚854  _runtime‚854C;F‹Äš¾ & lr-AdamW/pg1‚0.0004339980783986498 & lr-AdamW/pg2‚0.0004339980783986498 trainer/global_step‚999 % _timestamp‚1.7843583628364162e+09 _runtime‚ 854.911735764 _step‚78ètjț & lr-AdamW/pg1‚0.0004339980783986498 & lr-AdamW/pg2‚0.0004339980783986498 trainer/global_step‚999 % _timestamp‚1.7843583628364162e+09 _runtime‚ 854.911735764 _step‚78NaUŒH0œ+ _wandbruntime‚854  _runtime‚854ÇoÍ'ÇO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2147539 trainer/global_step‚999 _step‚79 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9226471 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34647402048000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34647500849152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1755918 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151136003915776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1049967425536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7877192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151135986876416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33584542992384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152186312679936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152185954301952 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151135986876416 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9236447 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9226471 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1048359829215 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7877192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1050308764160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1049967425536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7877194 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152186312679936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 $ _timestamp‚1.784358362837503e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150363282116608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1048700379101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2147259 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33584561604608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886Wd:YžËO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34647402048000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34647500849152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_seg @Õx$ments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1755918 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151136003915776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1049967425536 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7877192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151135986876416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33584542992384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152186312679936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152185954301952 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151135986876416 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9236447 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9226471 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1048359829215 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7877192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1050308764160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1049967425536 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7877194 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152186312679936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 $ _timestamp‚1.784358362837503e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150363282116608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1048700379101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2147259 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33584561604608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2147539 trainer/global_step‚999 _step‚79 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9226471 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736O39=¡’Ÿ‚(:‡ ÚËìÒ»Åëº gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x›ì2à‚(:¸ ÚËìÒÇ Ÿ½ disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚276447$ disk./.usageGB‚586.3884925842285 disk.sda3.out‚ 21.88671875 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 21.88671875( proc.memory.availableMB‚ 248.63671875 disk.sda.out‚ 21.88671875 disk.dm-0.in‚ 1911.33984375 disk.rbd0.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 1911.33984375% disk.nvme0n1.out‚2407.33349609375 cpu‚26.08137196109978 network.sent‚ 1.467451e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2715.74609375) disk./.usagePercent‚39.35302375004424 disk.sda1.out‚0" disk.dm-0.out‚2407.33349609375 disk.rbd1.out‚1.9375# disk.nvme0c0n1.in‚ 1911.33984375$ memory_percent‚98.48244190216064) proc.memory.percent‚16.57559871673584' disk.nvme0c0n1.out‚2407.33349609375ŠiÁa0¡+ _wandbruntime‚855  _runtime‚855µ”U²×&¢ÑM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151559504221415 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1049370882937 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 _step‚80 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151283227906048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9235523 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9235523 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151283244945408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150509775749120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884826 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1050980052480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884826 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1051321398272 trainer/global_step‚999 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1757666 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2149338 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3907254…ƒ.ÖÛ&£ÕM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33617990363136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1050980052480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884826 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1051321398272 trainer/global_step‚999 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1757666 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2149338 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3907254 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151559504221415 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1049370882937 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 _step‚80 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151283227906048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9235523 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9235523 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151283244945408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150509775749120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884826 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1114350887424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158498055397376 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4094227 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1099316866009 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1100662927360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159599059663360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157688815710208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36370247422976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1049 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1843171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157688798670848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8258794 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17939715 ˜œÓ'°ÍO OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8258794 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8258792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 894.791614541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1049 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1843171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157688798670848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8258794 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17939715 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101004265984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2251327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35255980918784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158788132576217 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158498055397376 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4094498 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35255999531008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1114350887424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158498055397376 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4094227 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1099316866009 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1100662927360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159599059663360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157688815710208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36370247422976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚84 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1049 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8266424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35289409677312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158935654301923 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159746954941952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 % _timestamp‚1.7843584033277023e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4098336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1844913 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_b@Â)FÌatch_end/requested_bytes.large_pool.allocated‚157835326382080 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35289428289536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ëãGÛÝ&³×M E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100327919843 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17966391 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1101675554304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158935296712565 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚84 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1049 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8266424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35289409677312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158935654301923 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159746954941952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 % _timestamp‚1.7843584033277023e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4098336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1844913 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157835326382080 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35289428289536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9699965 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2253143 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9699965 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9689975 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 _runtime‚ 895.403303036 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159747313327104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1099987369845 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17956399 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8266424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1151699767808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36927418845184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1151699767808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10145351 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166164282656725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 trainer/global_step‚1099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 934.447142015 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640394 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8640392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165860106878976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4286246 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1151358429184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38093078987264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165014332264448 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167011806646784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1151358429184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166163925067479 % _timestamp‚1.7843584423686485e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36927437457408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1149592803031 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508‰1k(öÂÒO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167011806646784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1151358429184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166163925067479 % _timestamp‚1.7843584423686485e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36927437457408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1149592803031 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165860089839616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2355541 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1151699767808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36927418845184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1151699767808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10145351 YEDeviceStatsMonitor.on_train_batch ª†Óâ_start/requested_bytes.all.allocated‚166164282656725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 trainer/global_step‚1099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 934.447142015 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640394 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8640392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165860106878976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4286246 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1151358429184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38093078987264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165014332264448WÄZ90Ã+ _wandbruntime‚935  _runtime‚9352TH“Þ&ÄØM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1150603856753 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1152712401920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1152712401920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1932166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8648026 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8648024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10144427 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166007347908608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2357878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166311446793073 trainer/global_step‚1099 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167160060310528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165160859975680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165160842936320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1166774396416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1152371056128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166007330869248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166311804382431 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167160060310528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166007330869248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1932163 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 935.055797899 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36960847603712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1166685826048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2357612 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843584429802933e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8648026 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10144427 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2’Î-â&ÅÜM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36960847603712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1166685826048 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2357612 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843584429802933e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8648026 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10144427 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1150603856753 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1152712401920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1152712401920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1932166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8648026 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8648024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10144427 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166007347908608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2357878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166311446793073 trainer/global_step‚1099 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167160060310528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165160859975680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165160842936320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1166774396416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1152371056128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166007330869248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166311804382431 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167160060310528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166007330869248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1932163 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 935.055797899 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360Xtzc.0Æ+ _wandbruntime‚935  _runtime‚935j ÂþÇø -loss/FRt2/detection_loss‚1.51221480154932 1loss/SRqqt1/detection_loss‚1.1582494870022375 /loss/FBt2/detection_loss‚0.6703811760675735 epoch‚2 _step‚89 1loss/SRqqt2/assignment_loss‚0.702196012336048 1loss/FBt2/assignment_loss‚0.18842079301628958 1loss/SRqqt2/detection_loss‚0.7322019096400391 .loss/FBt1/detection_loss‚1.430678242444992 %loss/total_loss‚12.69512505667172 trainer/global_step‚1099 /loss/FRt1/detection_loss‚0.6454123507445872 /loss/FRt1/assignment_loss‚2.964298379394802 0loss/FRt2/assignment_loss‚1.4645564198729084 1loss/SRqqt1/assignment_loss‚0.887109406816971 0loss/FBt1/assignment_loss‚0.3394060777859522 % _timestamp‚1.7843584429833102e+09 _runtime‚ 935.058380459œj¦'Èü 1loss/SRqqt2/assignment_loss‚0.702196012336048 1loss/FBt2/assignment_loss‚0.18842079301628958 1loss/SRqqt2/detection_loss‚0.7322019096400391 .loss/FBt1/detection_loss‚1.430678242444992 %loss/total_loss‚12.69512505667172 trainer/global_step‚1099 /loss/FRt1/detection_loss‚0.6454123507445872 /loss/FRt1/assignment_loss‚2.964298379394802 0loss/FRt2/assignment_loss‚1.4645564198729084 1loss/SRqqt1/assignment_loss‚0.887109406816971 0loss/FBt1/assignment_loss‚0.3394060777859522 % _timestamp‚1.7843584429833102e+09 _runtime‚ 935.058380459 -loss/FRt2/detection_loss‚1.51221480154932 1loss/SRqqt1/detection_loss‚1.1582494870022375 /loss/FBt2/detection_loss‚0.6703811760675735 epoch‚2 _step‚89Yó@—Ä’É‚(:‡ ´ÌìÒø¼ª¼ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QVºÊ‚(:¯ ´ÌìÒª¸ë¼ disk.drbd2165.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚2439.44189453125 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2180.58984375" disk.dm-0.out‚2439.44189453125 proc.memory.rssMB‚2715.75$ disk./.usageGB‚586.3914909362793 disk.rbd1.in‚0 disk.rbd1.out‚2.125 disk.sda1.out‚0 cpu‚26.047533443310254 proc.cpu.threads‚17 network.recv‚299401) disk./.usagePercent‚39.35322497196615 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 2180.58984375 disk.sda3.out‚ 24.4453125( proc.memory.availableMB‚ 247.98828125 disk.sda2.in‚0 disk.dm-1.out‚ 24.4453125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2439.44189453125! disk.nvme0n1.in‚ 2180.58984375 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.percent‚16.57562255859375 disk.drbd2165.out‚0 network.sent‚ 1.648661e+06 disk.sda.out‚ 24.4453125$ memory_percent‚98.48639965057373(-Ö‘Ë‚(:† ÃÌìÒž±í» gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚234.06' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÈìï­Ì‚(:¢ ÃÌìҨţ¼ disk.dm-1.out‚24.75 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 2180.58984375& disk.nvme0c0n1.out‚2442.2236328125 disk.dm-1.in‚0) proc.memory.percent‚16.57564640045166 network.recv‚304108) disk./.usagePercent‚39.35321575569492! disk.nvme0n1.in‚ 2180.58984375 disk.sda.out‚24.75! disk.dm-0.out‚2442.2236328125$ memory_percent‚98.48482608795166 disk.sda2.in‚0# proc.memory.rssMB‚ 2715.75390625 cpu‚26.042776622049125 network.sent‚ 1.711713e+06 disk.rbd1.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚2442.2236328125 disk.sda.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 248.24609375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚2.25 disk.sda3.out‚24.75 disk.sda1.out‚0$ disk./.usageGB‚586.3913536071777 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 2180.58984375⮦â0Í+ _wandbruntime‚974  _runtime‚974Ð~\,ÇÎÁ trainer/global_step‚1149 % _timestamp‚1.7843584821817744e+09 _runtime‚ 974.257101368 _step‚90 ' lr-AdamW/pg1‚0.00043343425611936824 ' lr-AdamW/pg2‚0.00043343425611936824&óåšËÏÅ ' lr-AdamW/pg1‚0.00043343425611936824 ' lr-AdamW/pg2‚0.00043343425611936824 trainer/global_step‚1149 % _timestamp‚1.7843584821817744e+09 _runtime‚ 974.257101368 _step‚90ZípE)0Ð+ _wandbruntime‚974  _runtime‚974Ö†x ÑÍO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38598856771584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1202395269632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172339882897408 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172339865858048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19611819 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1202395269632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9021994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173222141321216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173222158360576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1202053931008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286912 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2459152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169{óFƒ×'ÒÑO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9021992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1217098020864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1200209289939 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10599803 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173222141321216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10599803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1202053931008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4476820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2017668 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173540432737233 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173222158360576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174424195252224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39815864117760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173540075147987 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1200549839825 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172339882897408 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172339865858048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19611819 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1202395269632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9021994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173222141321216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173222158360576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1202053931008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286912 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2459152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38598856771584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1202395269632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174572807293952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2019413 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1203407903744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173687596873581 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1203066557952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚92 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9029624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39850304879616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174572807293952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10608869 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39850418353664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2019416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173369399390208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173687954462939 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843584827956834e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9029624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174572448908800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2461208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚ 974.871250881 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1203407903744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10608869 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38632304142336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9029626¤39ô7ÕÜM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173369382350848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1203066557952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1218114211328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1218019349504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39850418353664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2019416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173369399390208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173687954462939 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843584827956834e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9029624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174572448908800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2461208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚ 974.871250881 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1203407903744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10608869 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38632304142336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9029626 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174572807293952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2019413 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1203407903744 SEDeviceStatsMonitor.on_train_ba¡+Æ«tch_end/allocated_bytes.small_pool.peak‚ 554757632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173687596873581 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1203066557952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚92 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9029624 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39850304879616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174572807293952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10608869 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 E2â- _wandbruntime‚1014  _runtime‚1014“èÔ'ãÎO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180584209842176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2104921 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11054255 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9403592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11054255 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179665416491008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4667712 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41538553682944 % _timestamp‚1.7843585220178509e+09 _step‚95 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253090771456 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9403594 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1250825776847 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180584192802816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9403594 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40270294697984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180584209842176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181837300613632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181836942235648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252749432832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1014.093349559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181837300613632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4667451 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984:FØ'äÒO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181837300613632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4667451 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180584209842176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2104921 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11054255 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9403592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11054255 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179665416491008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4667712 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41538553682944 % _timestamp‚1.7843585220178509e+09 _step‚95 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253090771456 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9403594 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1250825776847 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180584192802816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9403594 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40270294697984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180584209842176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181837300613632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181836942235648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252749432832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1014.093349559_¾©³$2å- _wandbruntime‚1014  _runtime‚1014ŒEþßæØM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180731450871808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253762059776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2106666 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11063321 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9411224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20464555 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179811927162880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStq‘¼ÿatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _step‚96 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181985195892224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11063321 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41573107226624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251836830569 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843585226305401e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11053331 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4671449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 trainer/global_step‚1199 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9411226 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254103405568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚1014.706137705 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11053331 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269365157888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179811910123520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998;zÛåâ&çÜM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2106666 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11063321 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9411224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20464555 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179811927162880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _step‚96 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181985195892224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11063321 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41573107226624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251836830569 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843585226305401e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11053331 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4671449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 trainer/global_step‚1199 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9411226 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254103405568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚1014.706137705 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11053331 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269365157888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179811910123520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180731450871808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253762059776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0`<]ÿZ2è- _wandbruntime‚1014  _runtime‚1014«! »éþ epoch‚2 trainer/global_step‚1199 2loss/SRqqt1/assignment_loss‚0.8634468400401861 1loss/FBt1/assignment_loss‚0.09018485316961744 /loss/FRt1/detection_loss‚0.6743697118454273 .loss/FRt2/detection_loss‚1.343639375631379 /loss/FBt2/detection_loss‚0.5702259557384315 % _timestamp‚1.7843585226329243e+09 _step‚97 /loss/FBt1/detection_loss‚1.0665630630683154 0loss/FRt2/assignment_loss‚1.3271370803926752 1loss/SRqqt2/assignment_loss‚0.618196458788893 1loss/FBt2/assignment_loss‚0.16892802714328806 &loss/total_loss‚11.113061877342552 _runtime‚1014.707998726 /loss/FRt1/assignment_loss‚2.527559492394731 1loss/SRqqt1/detection_loss‚1.1384560104725616 1loss/SRqqt2/detection_loss‚0.7243550086570477^žê‚ 0loss/FRt2/assignment_loss‚1.3271370803926752 1loss/SRqqt2/assignment_loss‚0.618196458788893 1loss/FBt2/assignment_loss‚0.16892802714328806 &loss/total_loss‚11.113061877342552 _runtime‚1014.707998726 /loss/FRt1/assignment_loss‚2.527559492394731 1loss/SRqqt1/detection_loss‚1.1384560104725616 1loss/SRqqt2/detection_loss‚0.7243550086570477 epoch‚2 trainer/global_step‚1199 2loss/SRqqt1/assignment_loss‚0.8634468400401861 1loss/FBt1/assignment_loss‚0.09018485316961744 /loss/FRt1/detection_loss‚0.6743697118454273 .loss/FRt2/detection_loss‚1.343639375631379 /loss/FBt2/detection_loss‚0.5702259557384315 % _timestamp‚1.7843585226329243e+09 _step‚97 /loss/FBt1/detection_loss‚1.0665630630683154ay»i¼šë‚(: ÿÌìÒå劽 gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tw•Œ¹ì‚(:® ÿÌìÒÌõ‰¾ disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 2376.7265625* proc.memory.percent‚16.477179527282715 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 26.15234375# disk.nvme0n1.out‚2457.638671875 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.3125 disk.sda3.out‚ 26.15234375 disk.dm-0.out‚2457.638671875" memory_percent‚98.386549949646 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚ 2376.7265625 disk.sda.out‚ 26.15234375( proc.memory.availableMB‚ 264.34765625 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚2457.638671875 disk.nvme0n1.in‚ 2376.7265625 network.sent‚ 1.802464e+06) disk./.usagePercent‚39.35316071407506$ disk./.usageGB‚586.3905334472656 disk.dm-1.in‚0# proc.memory.rssMB‚ 2699.62109375 cpu‚26.04357718051156 network.recv‚319328¿Ÿæ>2í- _wandbruntime‚1032  _runtime‚1032¡öSô.î«{ *loss/val_total_loss‚11.658458282688272 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14567263424396515 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20544880628585815 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5131086111068726 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.16163618862628937 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6953892707824707 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48139145970344543 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46134844422340393 +jet/accuracy_1_of_1‚0.22247175872325897 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚1032.182137873 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6701694130897522 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9947651624679565 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22846344113349915 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6995465159416199 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5780181884765625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 *jet/accuracy_2_of_3‚0.4102074205875397 'particle/f_score‚0.6026144027709961 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2057729959487915 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09829481691122055 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4302045702934265 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5681775212287903 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11977776139974594 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4444444477558136 +particle/specificity‚0.8923502564430237 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5868452787399292 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2057729959487915 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.34221145510673523 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.24025356769561768 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9947651624679565 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 *jet/accuracy_1_of_3‚0.8919604420661926 epoch‚2 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47264671325683594 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9075071215629578 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019342968240380287 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13213615119457245 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5811559557914734 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9822224378585815 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.44065484404563904 (particle/accuracy‚0.8297874927520752 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5125728845596313 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3888055682182312 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2057190239429474 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8737878203392029 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01694791205227375 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4827878177165985 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7011338472366333 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10466693341732025 " _timestamp‚1.7843585401061e+09 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07875671982765198 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6415533423423767 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26233887672424316 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791269302368 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2698231339454651 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1060633659362793 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8167392015457153 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.16517511010169983 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_4‚0.7244560718536377 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23142503201961517 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.16655123233795166 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008329509757459164 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_2_of_2‚0.9998962879180908 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24351675808429718 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20225511491298676 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4617151618003845 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7251092195510864 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5314579010009766 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09144729375839233 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3074784576892853 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906565546989441 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848358631134033 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5771965980529785 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46328744292259216 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2496379166841507 ,jet/accuracy_4_of_4‚0.009931244887411594 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09072320908308029 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0249035581946373 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5356622338294983 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6543722152709961 +jet/accuracy_3_of_4‚0.06924250721931458 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5910724997520447 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09072320908308029 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6301323771476746 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.09590565413236618 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.20513924956321716 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9952581524848938 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46081414818763733 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893547892570496 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5375672578811646 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08714359998703003 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48139041662216187 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6543722152709961 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9855064153671265 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14132633805274963 particle/accuracy_1_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5120293498039246 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46119821071624756 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8154648542404175 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6094387769699097 *jet/accuracy_1_of_4‚0.9523743391036987 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9887386560440063 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4820833206176758 trainer/global_step‚1199 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9782036542892456 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926013946533203 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.587358295917511 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46081414818763733 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46099385619163513 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12813226878643036 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6101338863372803 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8510312438011169 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004202599171549082 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.07159361243247986 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26848936080932617 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2069103866815567 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009758632630109787 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/aß,º¦”ccuracy_1_of_3‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5212394595146179 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9622683525085449 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5354502201080322 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731480598449707 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4558430314064026 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.05187332630157471 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16176335513591766 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5170120000839233 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 5validation_average_jet_accuracy‚0.278409868478775 particle/accuracy_4_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16505283117294312 +particle/sensitivity‚0.6010119318962097 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7008314728736877 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6345518827438354 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959591627120972 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9644447565078735 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_3_of_4‚0.9944444298744202 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.514812171459198 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9783066511154175 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06954038143157959 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.513189971446991 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.07297931611537933 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.619771420955658 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_3‚0.9989922046661377 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990960955619812 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855620861053467 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5677463412284851 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18070633709430695 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5032258033752441 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9935126304626465 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10111945122480392 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_1_of_5‚1 _step‚98 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5386062860488892 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.20872735977172852 particle/accuracy_2_of_4‚1>\Ë=µ=ï¯{ 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.513189971446991 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.07297931611537933 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.619771420955658 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_3‚0.9989922046661377 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990960955619812 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9855620861053467 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5677463412284851 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18070633709430695 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5032258033752441 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9935126304626465 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.10111945122480392 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_1_of_5‚1 _step‚98 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5386062860488892 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.20872735977172852 particle/accuracy_2_of_4‚1 *loss/val_total_loss‚11.658458282688272 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14567263424396515 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20544880628585815 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5131086111068726 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.16163618862628937 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6953892707824707 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48139145970344543 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46134844422340393 +jet/accuracy_1_of_1‚0.22247175872325897 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚1032.182137873 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6701694130897522 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9947651624679565 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22846344113349915 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6995465159416199 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5780181884765625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 *jet/accuracy_2_of_3‚0.4102074205875397 'particle/f_score‚0.6026144027709961 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2057729959487915 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09829481691122055 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4302045702934265 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5681775212287903 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11977776139974594 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4444444477558136 +particle/specificity‚0.8923502564430237 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5868452787399292 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2057729959487915 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.34221145510673523 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.24025356769561768 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9947651624679565 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 *jet/accuracy_1_of_3‚0.8919604420661926 epoch‚2 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47264671325683594 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9075071215629578 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019342968240380287 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13213615119457245 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5811559557914734 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9822224378585815 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.44065484404563904 (particle/accuracy‚0.8297874927520752 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5125728845596313 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3888055682182312 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2057190239429474 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8737878203392029 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01694791205227375 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4827878177165985 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7011338472366333 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.10466693341732025 " _timestamp‚1.7843585401061e+09 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07875671982765198 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6415533423423767 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26233887672424316 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791269302368 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2698231339454651 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1060633659362793 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8167392015457153 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.16517511010169983 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 *jet/accuracy_2_of_4‚0.7244560718536377 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23142503201961517 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.16655123233795166 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008329509757459164 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_2_of_2‚0.9998962879180908 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24351675808429718 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20225511491298676 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4617151618003845 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7251092195510864 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5314579010009766 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09144729375839233 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3074784576892853 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906565546989441 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848358631134033 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5771965980529785 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46328744292259216 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2496379166841507 ,jet/accuracy_4_of_4‚0.009931244887411594 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09072320908308029 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0249035581946373 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5356622338294983 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6543722152709961 +jet/accuracy_3_of_4‚0.06924250721931458 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5910724997520447 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09072320908308029 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6301323771476746 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.09590565413236618 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.20513924956321716 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9952581524848938 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46081414818763733 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893547892570496 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5375672578811646 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08714359998703003 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48139041662216187 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6543722152709961 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9855064153671265 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14132633805274963 particle/accuracy_1_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5120293498039246 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46119821071624756 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8154648542404175 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6094387769699097 *jet/accuracy_1_of_4‚0.9523743391036987 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9887386560440063 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4820833206176758 trainer/global_step‚1199 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9782036542892456 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926013946533203 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.587358295917511 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46081414818763733 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46099385619163513 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12813226878643036 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6101338863372803 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1133333295583725 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8510312438011169 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004202599171549082 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.07159361243247986 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26848936080932617 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2069103866815567 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009758632630109787 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5212394595146179 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9622683525085449 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5354502201080322 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731480598449707 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4558430314064026 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.05187332630157471 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16176335513591766 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5170120000839233 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 5validation_average_jet_accuracy‚0.278409868478775 particle/accuracy_4_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16505283117294312 +particle/sensitivity‚0.6010119318962097 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7008314728736877 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6345518827438354 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959591627120972 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9644447565078735 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_3_of_4‚0.9944444298744202 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.514812171459198 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9783066511154175 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06954038143157959 ("Purity/1FRt1SRqqt2FBt/event_purity‚0b¿>—¸“ð‚(:ˆ ŽÍìÒ¬–¼ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚241.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.2168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãáþñ‚(:³ ŽÍìÒ˜Áõ¼ disk.dm-1.out‚ 26.4609375 disk.sda.out‚ 26.4609375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 2376.7265625 disk.sda1.out‚0 disk.rbd1.out‚2.375( proc.memory.availableMB‚ 497.49609375 cpu‚26.084081854378617 network.sent‚ 1.852353e+06 disk.sda.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚2707.51904296875$ disk./.usageGB‚586.6350708007812 disk.sda1.in‚0 disk.sda3.out‚ 26.4609375 disk.sda2.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2714.74609375 network.recv‚323629* disk./.usagePercent‚39.369571821049995' disk.nvme0c0n1.out‚2707.51904296875 disk.dm-0.in‚ 2376.7265625" disk.dm-0.out‚2707.51904296875 disk.drbd2165.out‚0$ memory_percent‚96.96352481842041) proc.memory.percent‚16.56949520111084 disk.rbd0.in‚0 disk.nvme0n1.in‚ 2376.7265625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0•ƒsÖ“ò‚(:ˆ ÍìÒòë» gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚206.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ<Š$­ó‚(:¢ ÍìÒ®ìÙ¼ disk.dm-1.out‚ 26.828125$ memory_percent‚98.47128391265869 cpu‚26.07896300486483" disk.nvme0n1.out‚ 2711.92578125 disk.sda2.in‚0 disk.dm-1.in‚0 network.sent‚ 1.909633e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 250.46484375) proc.memory.percent‚16.57562255859375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.out‚2.375 proc.memory.rssMB‚2715.75 disk.drbd2165.in‚0 disk.dm-0.in‚ 2483.17578125 disk.dm-0.out‚ 2711.92578125 disk.rbd1.in‚0 disk.sda3.out‚ 26.828125) disk./.usagePercent‚39.36964299114452$ disk./.usageGB‚586.6361312866211# disk.nvme0c0n1.in‚ 2483.17578125 disk.rbd0.in‚0 network.recv‚328100 disk.sda.out‚ 26.828125 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 2711.92578125! disk.nvme0n1.in‚ 2483.17578125øœ­ô‚(:¢ ¬ÍìÒÅæ» disk.dm-0.out‚2963.728515625 disk.dm-1.out‚ 27.1171875# memory_percent‚98.4705924987793( disk./.usagePercent‚39.3695562045904 disk.rbd1.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚2715.75 cpu‚26.07379032378531% disk.nvme0c0n1.out‚2963.728515625 disk.drbd2165.out‚0 disk.dm-0.in‚ 2483.17578125 disk.sda3.in‚0 disk.sda.out‚ 27.1171875 disk.rbd1.out‚2.4375 disk.rbd0.in‚0# disk.nvme0n1.out‚2963.728515625 disk.sda3.out‚ 27.1171875) proc.memory.percent‚16.57562255859375 network.recv‚331306# disk./.usageGB‚586.634838104248 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 2483.17578125! disk.nvme0n1.in‚ 2483.17578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 1.9146e+06 disk.sda.in‚0 disk.drbd2165.in‚0& proc.memory.availableMB‚ 250.578125 proc.cpu.threads‚17 disk.dm-1.in‚02Ñõ6“õ‚(:ˆ ¬ÍìÒàÍè¼ gpu.0.gpu‚57.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç“Ò)2ö- _wandbruntime‚1073  _runtime‚1073ÎãÈ÷ ' lr-AdamW/pg1‚0.00043298900917532657 ' lr-AdamW/pg2‚0.00043298900917532657 trainer/global_step‚1249 % _timestamp‚1.7843585815374186e+09 _runtime‚1073.612798276 _step‚99g†ú¬ÌøÆ trainer/global_step‚1249 % _timestamp‚1.7843585815374186e+09 _runtime‚1073.612798276 _step‚99 ' lr-AdamW/pg1‚0.00043298900917532657 ' lr-AdamW/pg2‚0.00043298900917532657c"?_’2ù- _wandbruntime‚1073  _runtime‚1073–ÃóX“ úÏO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189875827974144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1337441371648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚1249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2194688 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9905860 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2194691 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190219115837562 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319161988984 _step‚100 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1321519480832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1321178141696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905858 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2696078 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905860 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191196989076480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9905858 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188899613298688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41957644724224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚189875810934784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190218758248312 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11569056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚189875827974144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21474914 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1321519480832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1319502538874 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4890769 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2695820 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43295180702208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11579033 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191197347454976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41957663336448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191197347454976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41957663336448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1337441371648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚1249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2194688 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9905860 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2194691 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190219115837562 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319161988984 _step‚100 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1321519480832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1321178141696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905858 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2696078 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905860 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191196989076480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9905858 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188899613298688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41957644724224dŽ•Ô22ü- _wandbruntime‚1074  _runtime‚1074X€ Cà&ýÚM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913492 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43329635350528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 trainer/global_step‚1249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190023069003776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843585821506631e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1322190768640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11578108 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41991092094976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2196436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913490 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269Æv1 ä&þÞM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843585821506631e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1322190768640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11578108 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41991092094976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2196436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913490 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913492 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43329635350528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 trainer/global_step‚1249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190023069003776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872eúâMx2ÿ- _wandbruntime‚1074  _runtime‚10745Ú»Œ€€ 0loss/FRt1/assignment_loss‚2.5353353704542463 0loss/FRt2/assignment_loss‚1.3410473793207525 2loss/SRqqt1/assignment_loss‚0.7771125584351436 2loss/SRqqt2/assignment_loss‚0.6711008836689638 /loss/FBt1/detection_loss‚1.7336432244628668 trainer/global_step‚1249 0loss/FBt2/assignment_loss‚0.1797320827443898 1loss/SRqqt1/detection_loss‚1.1672390647358815 % _timestamp‚1.7843585821530929e+09 _runtime‚1074.228170963 /loss/FBt2/detection_loss‚0.7024662390088747 &loss/total_loss‚11.974769387075881 /loss/FRt1/detection_loss‚0.6674078313181115 /loss/FRt2/detection_loss‚1.4052876234707798 0los×}ÉÄzs/SRqqt2/detection_loss‚0.704303905260917 1loss/FBt1/assignment_loss‚0.09009322419495333 epoch‚3 _step‚102'¥² „ 0loss/FRt1/assignment_loss‚2.5353353704542463 0loss/FRt2/assignment_loss‚1.3410473793207525 2loss/SRqqt1/assignment_loss‚0.7771125584351436 2loss/SRqqt2/assignment_loss‚0.6711008836689638 /loss/FBt1/detection_loss‚1.7336432244628668 trainer/global_step‚1249 0loss/FBt2/assignment_loss‚0.1797320827443898 1loss/SRqqt1/detection_loss‚1.1672390647358815 % _timestamp‚1.7843585821530929e+09 _runtime‚1074.228170963 /loss/FBt2/detection_loss‚0.7024662390088747 &loss/total_loss‚11.974769387075881 /loss/FRt1/detection_loss‚0.6674078313181115 /loss/FRt2/detection_loss‚1.4052876234707798 0loss/SRqqt2/detection_loss‚0.704303905260917 1loss/FBt1/assignment_loss‚0.09009322419495333 epoch‚3 _step‚102f åj1‘‚‚(:† »ÍìÒœ»ž» gpu.0.gpu‚51.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_?ØI°ƒ‚(:¥ »ÍìÒ½Žˆ¼ disk.sda2.out‚0 proc.memory.rssMB‚2715.75 proc.cpu.threads‚17 network.recv‚334972# disk.nvme0c0n1.in‚ 2483.17578125 disk.sda.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 249.83984375$ disk./.usageGB‚586.6348037719727 disk.dm-0.out‚ 2966.89453125 disk.dm-1.in‚0 disk.rbd1.out‚2.4375! disk.nvme0n1.in‚ 2483.17578125 network.sent‚ 1.919402e+06$ disk.nvme0c0n1.out‚ 2966.89453125 disk.sda.out‚ 27.7421875 disk.sda2.in‚0 cpu‚26.06915261065556( disk./.usagePercent‚39.3695539005226 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.57562255859375 disk.rbd1.in‚0$ memory_percent‚98.47509860992432" disk.nvme0n1.out‚ 2966.89453125 disk.dm-0.in‚ 2483.17578125 disk.dm-1.out‚ 27.7421875 disk.sda3.out‚ 27.7421875 disk.sda1.in‚0 disk.sda1.out‚0탞ˑ„‚(:† ÊÍìҴƺº gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚192.53' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ©/Ǻ…‚(:¯ ÊÍìÒâÔ¹½) disk./.usagePercent‚39.36956183675616 disk.sda2.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚2972.57666015625) proc.memory.percent‚16.57562255859375 network.sent‚ 1.984159e+06 disk.dm-1.out‚ 28.33203125 disk.sda.in‚0 disk.sda3.out‚ 28.33203125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2483.17578125' disk.nvme0c0n1.out‚2972.57666015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 28.33203125$ memory_percent‚98.47395420074463( proc.memory.availableMB‚ 250.02734375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚2.5! disk.nvme0n1.in‚ 2483.17578125 disk.sda3.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚586.6349220275879 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚339234 disk.dm-0.in‚ 2483.17578125" disk.dm-0.out‚2972.57666015625 proc.memory.rssMB‚2715.75 cpu‚26.06436166834998¼†’­“†‚(:ˆ ÙÍìÒ•£†½ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜j_ ©‡‚(:ž ÙÍìÒõ‹¾ disk.drbd2165.in‚0 network.sent‚ 1.987336e+06 disk.dm-0.in‚ 2625.0234375 disk.dm-0.out‚ 2975.234375 disk.dm-1.out‚ 28.58984375 disk.sda.in‚0 disk.sda3.out‚ 28.58984375 disk.rbd1.out‚2.5 disk.rbd0.in‚0) proc.memory.percent‚16.57562255859375 cpu‚26.05951220015633( proc.memory.availableMB‚ 250.02734375 disk.rbd1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0n1.out‚ 2975.234375* disk./.usagePercent‚39.369557740635614" disk.nvme0c0n1.out‚ 2975.234375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 2625.0234375 disk.sda.out‚ 28.58984375 disk.sda1.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚2715.75$ memory_percent‚98.47395420074463 network.recv‚342940$ disk./.usageGB‚586.6348609924316" disk.nvme0c0n1.in‚ 2625.0234375 disk.sda3.in‚0 disk.sda1.in‚0ÂC¼P2ˆ- _wandbruntime‚1113  _runtime‚1113UpòÓĉ¾ % lr-AdamW/pg1‚0.000432745617636121 % lr-AdamW/pg2‚0.000432745617636121 trainer/global_step‚1299 % _timestamp‚1.7843586214006057e+09 _runtime‚ 1113.47711508 _step‚103ÔLlìÈŠÂ % lr-AdamW/pg1‚0.000432745617636121 % lr-AdamW/pg2‚0.000432745617636121 trainer/global_step‚1299 % _timestamp‚1.7843586214006057e+09 _runtime‚ 1113.47711508 _step‚103g¶4%2‹- _wandbruntime‚1113  _runtime‚1113ê`ó8Ú'ŒÔO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12033485 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197237879455744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1369778475892 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12023508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚1299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚198610094438400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196225129852928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196225146892288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197594908328820 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚197237862416384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5081097 _runtime‚1113.477206834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10287458 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚197237862416384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388655593984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843586214013832e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10287460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2281938 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43629101262848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1372214982656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1371873643520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197237879455744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2281941 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198609736059904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1371873643520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2799156 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10287460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2281938 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43629101262848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1372214982656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1371873643520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197237879455744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2281941 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198609736059904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1371873643520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2799156 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚1299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚198610094438400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196225129852928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196225146892288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197594908328820 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚197237862416384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438hž®º2Ž- _wandbruntime‚1114  _runtime‚1114#g˜jÞM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10295092 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197385103446016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196371657564160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198758348102144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2800946 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1372886270464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12042551 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 _step‚105 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1114.087849852 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198757989716480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚197385103446016 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2801198 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1389761909760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12032560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚1299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚197385120485376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1373227616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197742787643776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2283686 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196371640524800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1372886270464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197742430054414 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371130079616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 244066549761×Düè&âM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12032560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚1299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚197385120485376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1373227616768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197742787643776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2283686 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196371640524800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1372886270464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197742430054414 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371130079616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10295092 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197385103446016 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196371657564160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198758348102144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2800946 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1372886270464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12042551 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 _step‚105 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1114.087849852 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198757989716480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚197385103446016 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2801198 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1389761909760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439887425024 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45300539189248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2369191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45300520577024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10669060 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1420394962800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10669058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12477960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204599913897984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10669060 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46740408002048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12487937 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚206022841421824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚206022841421824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2902291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2369188 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422910484480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204599913897984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚1153.173675302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203550680485888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204599913897984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10669060 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46740408002048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12487937 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚206022841421824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚206022841421824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2902291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2369188 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422910484480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204599913897984 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚1153.173675302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203550680485888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439887425024 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45300539189248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2369191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45300520577024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10669060 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1420394962800lè°¤+2ž- _wandbruntime‚1153  _runtime‚1153šõ…ìä&ŸÞM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204747154927616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204747154927616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚109 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206171095085568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12487012 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205118580134922 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1423923118592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203697191157760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1349 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2904096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205118937724284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2370933 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚467748532367362‚â£è& âM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204747154927616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚109 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206171095085568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12487012 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205118580134922 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1423923118592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203697191157760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1349 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2904096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205118937724284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2370933 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46774853236736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204747154927616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976mË~˜â2¡- _wandbruntime‚1153  _runtime‚11533:¢9¢‚ trainer/global_step‚1349 /loss/FRt1/assignment_loss‚2.422186635655781 0loss/FRt2/assignment_loss‚1.3512125501603436 1loss/FBt2/assignment_loss‚0.11150157638749843 /loss/FRt1/detection_loss‚0.6493780506178587 /loss/FBt2/detection_loss‚0.5233618502975841 epoch‚3 % _timestamp‚1.7843586617135966e+09 2loss/SRqqt2/assignment_loss‚0.6368745663988158 1loss/SRqqt2/detection_loss‚0.7365860427396138 _runtime‚1153.788683828 2loss/SRqqt1/assignment_loss‚0.6510954668570775 2loss/FBt1/assignment_loss‚0.022435050008240864 /loss/FRt2/detection_loss‚1.3903811438373863 /loss/FBt1/detection_loss‚1.3884498316328973 &loss/total_loss‚11.030673475383171 _step‚110 1loss/SRqqt1/detection_loss‚1.1472107107900749ßÁôÑ £† _step‚110 1loss/SRqqt1/detection_loss‚1.1472107107900749 trainer/global_step‚1349 /loss/FRt1/assignment_loss‚2.422186635655781 0loss/FRt2/assignment_loss‚1.3512125501603436 1loss/FBt2/assignment_loss‚0.11150157638749843 /loss/FRt1/detection_loss‚0.6493780506178587 /loss/FBt2/detection_loss‚0.5233618502975841 epoch‚3 % _timestamp‚1.7843586617135966e+09 2loss/SRqqt2/assignment_loss‚0.6368745663988158 1loss/SRqqt2/detection_loss‚0.7365860427396138 _runtime‚1153.788683828 2loss/SRqqt1/assignment_loss‚0.6510954668570775 2loss/FBt1/assignment_loss‚0.022435050008240864 /loss/FRt2/detection_loss‚1.3903811438373863 /loss/FBt1/detection_loss‚1.3884498316328973 &loss/total_loss‚11.030673475383171nØT**¤‚(:’ †ÎìǪ̀þ» gpu.0.gpu‚28.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HØäì·¥‚(:¬ †ÎìÒì÷μ disk.rbd1.out‚2.6875) proc.memory.percent‚16.57564640045166) disk./.usagePercent‚39.36959870184109$ disk./.usageGB‚586.6354713439941$ disk.nvme0c0n1.out‚ 2988.54296875 disk.rbd0.out‚0 network.sent‚ 2.059884e+06 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.out‚ 2988.54296875 network.recv‚354295" disk.nvme0c0n1.in‚ 2625.0234375" disk.nvme0n1.out‚ 2988.54296875 disk.sda1.out‚0$ memory_percent‚98.47147464752197# proc.memory.rssMB‚ 2715.75390625 cpu‚26.04671106162055 disk.nvme0n1.in‚ 2625.0234375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 29.93359375 disk.rbd0.in‚0( proc.memory.availableMB‚ 250.43359375 proc.cpu.threads‚17 disk.sda3.out‚ 29.93359375 disk.dm-0.in‚ 2625.0234375 disk.rbd1.in‚0 disk.sda.out‚ 29.93359375f¥Ã‹‘¦‚(:† •ÎìÒñÐƺ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™8Z*¾§‚(:³ •ÎìÒÔž§¾ disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚98.47171306610107( proc.memory.availableMB‚ 250.39453125 disk.dm-0.in‚ 2700.76171875# disk.nvme0c0n1.in‚ 2700.76171875$ disk.nvme0n1.out‚2992.4287109375 disk.sda2.in‚0! disk.dm-0.out‚2992.4287109375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.out‚2.6875 cpu‚26.042489718083253 network.sent‚ 2.122885e+06) disk./.usagePercent‚39.36956798093698$ disk./.usageGB‚586.6350135803223 disk.dm-1.out‚ 30.1796875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2992.4287109375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚358820 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 2700.76171875 disk.sda.out‚ 30.1796875 disk.sda3.out‚ 30.1796875# proc.memory.rssMB‚ 2715.75390625) proc.memory.percent‚16.57564640045166 proc.cpu.threads‚17vµ>÷ž¨‚(:“ ¤Îìқᬻ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.806' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.361200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ecÓÀ©‚(:µ ¤ÎìÒü׃½$ memory_percent‚98.47168922424316 cpu‚26.03850506821774' proc.memory.availableMB‚ 250.3984375 network.sent‚ 2.129826e+06 disk.rbd0.in‚0' disk.nvme0c0n1.out‚2997.12060546875 disk.sda.out‚ 30.65234375 disk.sda3.out‚ 30.65234375# proc.memory.rssMB‚ 2715.75390625! disk.nvme0n1.in‚ 2700.76171875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.75# disk.nvme0c0n1.in‚ 2700.76171875 disk.sda3.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚362211 disk.sda.in‚0 disk.sda1.out‚0" disk.dm-0.out‚2997.12060546875 disk.dm-1.out‚ 30.65234375) disk./.usagePercent‚39.36956183675616$ disk./.usageGB‚586.6349220275879 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 2700.76171875% disk.nvme0n1.out‚2997.12060546875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0) proc.memory.percent‚16.57564640045166`&X’2ª- _wandbruntime‚1193  _runtime‚1193ýÅ»[É«Ã _runtime‚1193.040652668 _step‚111 ' lr-AdamW/pg1‚0.00043221737658525494 ' lr-AdamW/pg2‚0.00043221737658525494 trainer/global_step‚1399 % _timestamp‚1.7843587009653304e+09éÂÍ¬Ç ' lr-AdamW/pg1‚0.00043221737658525494 ' lr-AdamW/pg2‚0.00043221737658525494 trainer/global_step‚1399 % _timestamp‚1.7843587009653304e+09 _runtime‚1193.040652668 _step‚111o›ç2ˆ2­- _wandbruntime‚1193  _runtime‚1193v&vÚ÷®ÔO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213435588405248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471011449708 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1473264647168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocat8w‚ã$ion.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1471351999598 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11050658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12942389 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210876197040128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1473605986304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12942389 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12932412 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210876214079488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11050660 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48463031559168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211961982418944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1473264647168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3005521 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211961965379584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7843587009664204e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212347208489836 _runtime‚1193.041893841 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213435588405248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1491073055744 trainer/global_step‚1399 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169(Þ'¯ØO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213435588405248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1491073055744 trainer/global_step‚1399 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213435588405248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471011449708 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1473264647168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1471351999598 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11050658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12942389 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210876197040128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1473605986304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12942389 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12932412 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210876214079488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11050660 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48463031559168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211961982418944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1473264647168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3005521 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211961965379584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7843587009664204e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212347208489836 _runtime‚1193.041893841pdÏÁn2°- _wandbruntime‚1193  _runtime‚1193—& pä&±ÞM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12951455 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474618620416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843587015800264e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48497587867136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11058290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212494730215430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12941464 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5465755 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474618620416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212109206409216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚113 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11058292 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47005405874176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213583842068992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11058290 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1492181992960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007569 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1492095520256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2458186 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48497482782208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211022724751360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1472363053432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212495087804792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47005387261952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212109223448576)€ ²âM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12941464 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5465755 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474618620416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212109206409216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonÞ¿}Qâitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚113 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11058292 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47005405874176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213583842068992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11058290 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1492181992960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007569 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1492095520256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2458186 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48497482782208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211022724751360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1472363053432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212495087804792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47005387261952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212109223448576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12951455 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474618620416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843587015800264e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48497587867136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11058290 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212494730215430q¸X$2³- _wandbruntime‚1193  _runtime‚1193ÄÜ䶴ÿ /loss/FBt1/detection_loss‚1.3177170394496485 trainer/global_step‚1399 /loss/FRt1/assignment_loss‚2.331779019170971 &loss/total_loss‚11.632175773541206 % _timestamp‚1.7843587015817583e+09 _runtime‚1193.656795234 _step‚114 1loss/SRqqt1/assignment_loss‚0.715312113671656 .loss/FRt2/detection_loss‚1.692213457301597 1loss/SRqqt1/detection_loss‚1.1563299693779585 1loss/SRqqt2/detection_loss‚0.7664948704160444 /loss/FBt2/detection_loss‚1.2498498503913187 0loss/FRt2/assignment_loss‚0.9525907778956182 2loss/SRqqt2/assignment_loss‚0.6620228747983095 1loss/FBt2/assignment_loss‚0.08268541691883502 epoch‚3 1loss/FBt1/assignment_loss‚0.06423100971949225 /loss/FRt1/detection_loss‚0.6409493744297569Š+„µ µƒ 1loss/SRqqt1/detection_loss‚1.1563299693779585 1loss/SRqqt2/detection_loss‚0.7664948704160444 /loss/FBt2/detection_loss‚1.2498498503913187 0loss/FRt2/assignment_loss‚0.9525907778956182 2loss/SRqqt2/assignment_loss‚0.6620228747983095 1loss/FBt2/assignment_loss‚0.08268541691883502 epoch‚3 1loss/FBt1/assignment_loss‚0.06423100971949225 /loss/FRt1/detection_loss‚0.6409493744297569 /loss/FBt1/detection_loss‚1.3177170394496485 trainer/global_step‚1399 /loss/FRt1/assignment_loss‚2.331779019170971 &loss/total_loss‚11.632175773541206 % _timestamp‚1.7843587015817583e+09 _runtime‚1193.656795234 _step‚114 1loss/SRqqt1/assignment_loss‚0.715312113671656 .loss/FRt2/detection_loss‚1.692213457301597r{§1š“¶‚(:ˆ ³ÎìÒ†ÿ™½ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñq ƒ»·‚(:° ³ÎìÒÓò‹¿ disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 2700.76171875) proc.memory.percent‚16.57564640045166 proc.cpu.threads‚17) disk./.usagePercent‚39.40384201361354 disk.sda.out‚ 30.859375 cpu‚26.034266204842574 disk.sda3.out‚ 30.859375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚3000.69287109375 network.recv‚365467 disk.sda3.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚3000.69287109375! disk.nvme0n1.in‚ 2700.76171875$ memory_percent‚98.47314357757568 network.sent‚ 2.14589e+06$ disk./.usageGB‚587.1457214355469 disk.sda1.in‚0 disk.rbd1.out‚2.75 disk.dm-1.out‚ 30.859375 disk.dm-0.in‚ 2700.76171875% disk.nvme0n1.out‚3000.69287109375( proc.memory.availableMB‚ 250.16015625# proc.memory.rssMB‚ 2715.75390625yC㑸‚(:† ÂÎìÒÃ×ຠgpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚261.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†9e¹Ä¹‚(:¹ ÂÎìÒù“’¾ disk.rbd1.out‚2.8125 disk.rbd0.in‚0% disk.nvme0n1.out‚3820.18603515625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 2.195806e+06 network.recv‚369904) disk./.usagePercent‚39.42286670150554$ disk./.usageGB‚587.4292030334473 disk.dm-0.in‚ 2700.76171875 disk.dm-1.in‚0 disk.dm-1.out‚ 31.29296875! disk.nvme0n1.in‚ 2700.76171875( proc.memory.availableMB‚ 250.47265625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 31.29296875) proc.memory.percent‚16.57564640045166 disk.sda1.out‚0# proc.memory.rssMB‚ 2715.75390625 cpu‚26.030464299558467 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2700.76171875' disk.nvme0c0n1.out‚3820.18603515625$ memory_percent‚98.47123622894287" disk.dm-0.out‚3820.18603515625 disk.sda.out‚ 31.29296875 disk.sda2.in‚0 proc.cpu.threads‚17Ã¥õ“º‚(:ˆ ÑÎìÒ¨­ò¼ gpu.0.gpu‚62.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iý »»‚(:° ÑÎìÒüˆô¾# disk.nvme0c0n1.in‚ 2764.40234375 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚39.42099016627956 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2715.75390625 cpu‚26.02687772611286 disk.sda1.out‚0 disk.dm-0.in‚ 2764.40234375" disk.dm-0.out‚3823.08154296875) proc.memory.percent‚16.57564640045166$ disk./.usageGB‚587.4012413024902 disk.rbd1.out‚2.8125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚3823.08154296875% disk.nvme0n1.out‚3823.08154296875 disk.sda3.out‚ 31.640625 disk.sda1.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 250.140625 proc.cpu.threads‚17 network.recv‚373218! disk.nvme0n1.in‚ 2764.40234375 disk.sda.in‚0 disk.sda.out‚ 31.640625 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 31.640625$ memory_percent‚98.47326278686523 network.sent‚ 2.200733e+06 disk.rbd0.out‚0½ìR2¼- _wandbruntime‚1232  _runtime‚1232:ö ý½ $ lr-AdamW/pg1‚0.00043193256100153 $ lr-AdamW/pg2‚0.00043193256100153 trainer/global_step‚1449 % _timestamp‚1.7843587406711302e+09 _runtime‚1232.746518538 _step‚115„P&'ǾÁ $ lr-AdamW/pg1‚0.00043193256100153 $ lr-AdamW/pg2‚0.00043193256100153 trainer/global_step‚1449 % _timestamp‚1.7843587406711302e+09 _runtime‚1232.746518538 _step‚115sÉ›ËÇ2¿- _wandbruntime‚1232  _runtime‚1232÷cØÚ'ÀÔO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843587406722786e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1521968486506 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219324016861184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5652004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1542380711424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1542285842944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13396841 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219324033900544 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48643415042048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50185682272768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48643396429824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219324033900544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220848335388672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11432258 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1232.747740554 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1523960148992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1524301488128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218201747673088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220847977010176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13386864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1524301488128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11432260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219723358570344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13386864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2àõ›µ$ÁØO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1524301488128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11432260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219723358570344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13386864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843587406722786e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1521968486506 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219324016861184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5652004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1542380711424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1542285842944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13396841 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219324033900544 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48643415042048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50185682272768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48643396429824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219324033900544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220848335388672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11432258 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1232.747740554 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1523960148992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1524301488128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218201747673088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220847977010176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13386864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2545433 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13405907 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚1449 _step‚117 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48676825188352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843587412847695e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2545436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2545433 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13405907 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚1449 _step‚117 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48676825188352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2u‘¼PÌ2Å- _wandbruntime‚1233  _runtime‚1233ГoÆ‚ 0loss/FRt1/assignment_loss‚2.1473800267643925 0loss/FRt2/assignment_loss‚1.0106312209909611 1loss/SRqqt1/detection_loss‚1.2283222464404322 /loss/FBt1/detection_loss‚1.0913706220410488 2loss/SRqqt2/assignment_loss‚0.6839873920211775 1loss/FBt1/assignment_loss‚0.33611714669650944 /loss/FRt1/detection_loss‚0.6806494835673887 1loss/SRqqt2/detection_loss‚0.7577411962053878 &loss/total_loss‚11.120877563074309 epoch‚3 _runtime‚1233.362075872 _step‚118 /loss/FRt2/detection_loss‚1.7321256618851542 /loss/FBt2/detection_loss‚0.6620078317280179 trainer/global_step‚1449 2loss/SRqqt1/assignment_loss‚0.6739473376507786 1loss/FBt2/assignment_loss‚0.11659739708306063 % _timestamp‚1.7843587412870097e+09¥;& dž 2loss/SRqqt1/assignment_loss‚0.6739473376507786 1loss/FBt2/assignment_loss‚0.11659739708306063 % _timestamp‚1.7843587412870097e+09 0loss/FRt1/assignment_loss‚2.1473800267643925 0loss/FRt2/assignment_loss‚1.0106312209909611 1loss/SRqqt1/detection_loss‚1.2283222464404322 /loss/FBt1/detection_loss‚1.0913706220410488 2loss/SRqqt2/assignment_loss‚0.6839873920211775 1loss/FBt1/assignment_loss‚0.33611714669650944 /loss/FRt1/detection_loss‚0.6806494835673887 1loss/SRqqt2/detection_loss‚0.7577411962053878 &loss/total_loss‚11.120877563074309 epoch‚3 _runtime‚1233.362075872 _step‚118 /loss/FRt2/detection_loss‚1.7321256618851542 /loss/FBt2/detection_loss‚0.6620078317280179 trainer/global_step‚1449vh ËA“È‚(:ˆ àÎìÒ¹·Ò» gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%‰ÿ,½É‚(:² àÎìÒ‘¿™¼ disk.dm-1.out‚ 31.93359375* disk./.usagePercent‚39.373095764758816# proc.memory.rssMB‚ 2715.75390625! disk.nvme0n1.in‚ 2764.40234375 disk.sda2.out‚0 disk.dm-0.out‚3827.291015625# memory_percent‚98.4736680984497 network.sent‚ 2.205672e+06$ disk./.usageGB‚586.6875801086426 disk.rbd1.out‚2.875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚3827.291015625 disk.sda2.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 250.07421875) proc.memory.percent‚16.57564640045166% disk.nvme0c0n1.out‚3827.291015625 disk.sda.out‚ 31.93359375 cpu‚26.023131087853603 proc.cpu.threads‚17 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 2764.40234375 disk.sda.in‚0 disk.dm-1.in‚0 network.recv‚376458 disk.sda3.in‚0 disk.sda3.out‚ 31.93359375 disk.sda1.out‚0 disk.dm-0.in‚ 2764.402343752¤hq“Ê‚(:ˆ ïÎìұü gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&zm¨¾Ë‚(:³ ïÎìÒÈ»¾% disk.nvme0n1.out‚3829.47900390625 disk.dm-1.out‚ 32.2578125 disk.rbd1.out‚2.875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 2764.40234375 disk.sda.out‚ 32.2578125 network.sent‚ 2.27052e+06 disk.sda2.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚39.37136566584242 disk.sda.in‚0# proc.memory.rssMB‚ 2715.75390625) proc.memory.percent‚16.57564640045166 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚3829.47900390625 disk.sda3.in‚0 disk.sda3.out‚ 32.2578125$ memory_percent‚98.47238063812256 network.recv‚380819 disk.dm-0.in‚ 2764.40234375 disk.dm-1.in‚0( proc.memory.availableMB‚ 250.28515625$ disk./.usageGB‚586.6618003845215 disk.sda1.in‚0 cpu‚26.01965073768224 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚3829.47900390625! disk.nvme0n1.in‚ 2764.40234375ó4†2Ì- _wandbruntime‚1272  _runtime‚1272Ô¨ñUÈÍ trainer/global_step‚1499 $ _timestamp‚1.784358780592465e+09 _runtime‚1272.667776259 _step‚119 ' lr-AdamW/pg1‚0.00043163397280791957 ' lr-AdamW/pg2‚0.00043163397280791957ÜÊKÌÎÆ ' lr-AdamW/pg1‚0.00043163397280791957 ' lr-AdamW/pg2‚0.00043163397280791957 trainer/global_step‚1499 $ _timestamp‚1.784358780592465e+09 _runtime‚1272.667776259 _step‚119w‡CƒÌ2Ï- _wandbruntime‚1272  _runtime‚1272ó­âåŠÐÔO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1572244423524 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1574996989952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1574655650816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13851293 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚120 WEDeviceStatsMonitor.on_train_¥…É×P batch_start/active_bytes.small_pool.freed‚ 1574655650816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226686085382144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1593525947392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5842734 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228261082372096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11813860 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7843587805936656e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2630941 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228260723993600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25665153 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13841316 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11813860 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1272.669198585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872’5áÞ'ÑØO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1574655650816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226686085382144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1593525947392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5842734 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228261082372096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11813860 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7843587805936656e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2630941 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228260723993600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25665153 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13841316 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11813860 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1272.669198585 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1572244423524 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1574996989952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1574655650816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13851293 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50348281726976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11821492 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226833309372416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13860359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13850368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2632683 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11821492 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228409336035840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11821490 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 $ _timestamp‚1.784358781204025e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13860359 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228409336035840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3213868 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3213599 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13850368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227247030376446 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1576009624064 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2632686 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1594543176192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226833326411776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226833326411776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304XÜ…ìš$ÔáM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1576009624064 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50348281726976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11821492 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226833309372416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13860359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13850368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2632683 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11821492 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228409336035840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11821490 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 $ _timestamp‚1.784358781204025e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13860359 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228409336035840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3213868 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3213599 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13850368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227247030376446 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1576009624064 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UIDeviceStatsMonitor.on_train_batch_end/iûÐHsMnactive_split.large_pool.allocated‚2632686 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1594543176192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226833326411776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226833326411776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304yS¡z"2Õ- _wandbruntime‚1273  _runtime‚1273zŠšQÖ 1loss/SRqqt1/detection_loss‚1.0388524747955112 1loss/SRqqt2/detection_loss‚0.7119864332541409 /loss/FBt1/detection_loss‚1.5398756616881915 _runtime‚1273.282179416 _step‚122 0loss/FRt1/assignment_loss‚2.2654077143524773 2loss/SRqqt2/assignment_loss‚0.6352913380167189 .loss/FRt1/detection_loss‚0.630241565660981 &loss/total_loss‚10.919423597106096 2loss/SRqqt1/assignment_loss‚0.6784269787595848 2loss/FBt1/assignment_loss‚0.027731578877640977 trainer/global_step‚1499 % _timestamp‚1.7843587812071033e+09 0loss/FRt2/assignment_loss‚0.9619505273125996 1loss/FBt2/assignment_loss‚0.13213258275039408 .loss/FBt2/detection_loss‚0.536073142399663 epoch‚3 /loss/FRt2/detection_loss‚1.7614535992381937¦v" ×… 2loss/FBt1/assignment_loss‚0.027731578877640977 trainer/global_step‚1499 % _timestamp‚1.7843587812071033e+09 0loss/FRt2/assignment_loss‚0.9619505273125996 1loss/FBt2/assignment_loss‚0.13213258275039408 .loss/FBt2/detection_loss‚0.536073142399663 epoch‚3 /loss/FRt2/detection_loss‚1.7614535992381937 1loss/SRqqt1/detection_loss‚1.0388524747955112 1loss/SRqqt2/detection_loss‚0.7119864332541409 /loss/FBt1/detection_loss‚1.5398756616881915 _runtime‚1273.282179416 _step‚122 0loss/FRt1/assignment_loss‚2.2654077143524773 2loss/SRqqt2/assignment_loss‚0.6352913380167189 .loss/FRt1/detection_loss‚0.630241565660981 &loss/total_loss‚10.919423597106096 2loss/SRqqt1/assignment_loss‚0.6784269787595848z V'nžØ‚(:“ þÎìÒÜ¥Ò» gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚153.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.631200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à ¹Ù‚(:® þÎìÒïÐ×¼ disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 32.515625 disk.rbd0.in‚0! disk.nvme0n1.in‚ 2764.40234375 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 2764.40234375 disk.rbd1.in‚0* disk./.usagePercent‚39.371383074354746' disk.nvme0c0n1.out‚3834.98974609375 disk.drbd2165.in‚0 disk.rbd1.out‚3# proc.memory.rssMB‚ 2715.75390625) proc.memory.percent‚16.57564640045166 cpu‚26.016227374167816 network.recv‚383904 disk.rbd0.out‚0 disk.sda1.in‚0" disk.dm-0.out‚3834.98974609375$ memory_percent‚98.47378730773926' proc.memory.availableMB‚ 250.0546875 proc.cpu.threads‚17 network.sent‚ 2.273442e+06$ disk./.usageGB‚586.6620597839355# disk.nvme0c0n1.in‚ 2764.40234375% disk.nvme0n1.out‚3834.98974609375 disk.sda.out‚ 32.515625 disk.sda3.out‚ 32.515625š-)HžÚ‚(:“ ÏìÒ¨‘öº gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.314' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.062799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vû®Û‚(:£ ÏìÒú´ã¼ network.sent‚ 2.336252e+06! disk.nvme0c0n1.in‚ 2771.109375 disk.sda1.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚3837.8271484375! disk.dm-0.out‚3837.8271484375 disk.rbd0.in‚0 disk.sda3.out‚ 32.890625 disk.dm-0.in‚ 2771.109375 disk.nvme0n1.in‚ 2771.109375 disk.sda3.in‚0$ memory_percent‚98.47569465637207) proc.memory.percent‚16.57564640045166 proc.cpu.threads‚17# disk./.usageGB‚586.662036895752 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚3 disk.rbd0.out‚0& disk.nvme0c0n1.out‚3837.8271484375 disk.sda.in‚0 network.recv‚388431) disk./.usagePercent‚39.37138153830954 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 32.890625 cpu‚26.012614733984773 disk.sda2.out‚0 disk.sda.out‚ 32.890625' proc.memory.availableMB‚ 249.7421875# proc.memory.rssMB‚ 2715.75390625 disk.drbd2165.in‚0œ†MÂÀÜ‚(:µ œÏìÒìÃ༠disk.rbd1.in‚0! disk.nvme0n1.in‚ 2861.05859375% disk.nvme0n1.out‚3843.62939453125& proc.memory.availableMB‚ 249.984375 proc.cpu.threads‚17 network.recv‚391074 disk.rbd1.out‚3.0625 cpu‚26.009276669100853$ memory_percent‚98.47421646118164 disk.sda.in‚0* disk./.usagePercent‚39.370991126819824$ disk./.usageGB‚586.6562194824219 disk.rbd0.out‚0 disk.sda3.out‚ 33.2421875 disk.sda1.in‚0 disk.dm-0.in‚ 2861.05859375 disk.sda.out‚ 33.2421875 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.57564640045166 network.sent‚ 2.340276e+06# proc.memory.rssMB‚ 2715.75390625 disk.sda2.in‚0" disk.dm-0.out‚3843.62939453125 disk.dm-1.out‚ 33.2421875# disk.nvme0c0n1.in‚ 2861.05859375 disk.sda3.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚3843.62939453125 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0KL‹Ý“Ý‚(:ˆ œÏìÒ‚‹Ó¼ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ém2Þ- _wandbruntime‚1312  _runtime‚1312ï?»EÈß ' lr-AdamW/pg1‚0.00043132163118218527 ' lr-AdamW/pg2‚0.00043132163118218527 trainer/global_step‚1549 $ _timestamp‚1.784358820246951e+09 _runtime‚1312.322280357 _step‚1235å":ÌàÆ trainer/global_step‚1549 $ _timestamp‚1.784358820246951e+09 _runtime‚1312.322280357 _step‚123 ' lr-AdamW/pg1‚0.00043132163118218527 ' lr-AdamW/pg2‚0.00043132163118218527{}ÒÕ¤2á- _wandbruntime‚1312  _runtime‚1312Fg8MÚ'âÔO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12195460 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3315648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53630858041856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232852814860288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1312.323546247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234048136863744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6033571 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51986290894848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622860910432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14295768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14305745 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1644670141952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12195458 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234048136863744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623201460322 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14295768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚124 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1625351152640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1625692491776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234048119824384 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1625692491776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2718191 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51986272282624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232852797820928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1625692491776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2718191 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51986272282624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232852797820928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1312.323546247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234048136863744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6033571 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51986290894848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622860910432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14295768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14305745 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1644670141952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12195458 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234048136863744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623201460322 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14295768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚124 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496|1@i2ä- _wandbruntime‚1313  _runtime‚1313›™âåßM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53665413956608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235821724633600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1626363779584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12203090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1626705125888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1626705125888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3317408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232999308492800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52019719653376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _step‚125 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3317690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12203090 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14314811 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_po—¯„­Ìol.freed‚ 1645593150464 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14304820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1549 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234195377893376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234195360854016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234195360854016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12203092 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12203092 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235822083019264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843588210462544e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26517903 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1313.121798434 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645593150464 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14304820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1549 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234195377893376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234195360854016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234195360854016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12203092 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12203092 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235822083019264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843588210462544e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26517903 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1313.121798434 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235821724633600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1626363779584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12203090 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1626705125888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1626705125888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3317408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232999308492800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52019719653376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _step‚125 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3317690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983}E7‡#2ç- _wandbruntime‚1313  _runtime‚1313u?Ͼè€ 2loss/SRqqt2/assignment_loss‚0.6384726510191279 /loss/FRt1/detection_loss‚0.6520068438625943 /loss/FBt1/detection_loss‚1.1755424976348876 &loss/total_loss‚11.018541922971103 % _timestamp‚1.7843588210492125e+09 /loss/FRt1/assignment_loss‚2.215674236173474 1loss/FBt2/assignment_loss‚0.19906261337480372 0loss/SRqqt1/detection_loss‚1.118236096139395 /loss/FBt2/detection_loss‚0.6743463542958159 _step‚126 2loss/SRqqt1/assignment_loss‚0.7259274262215684 trainer/global_step‚1549 epoch‚3 0loss/FRt2/assignment_loss‚0.9913539078060069 1loss/FBt1/assignment_loss‚0.19879630198686624 /loss/FRt2/detection_loss‚1.6614605122690493 1loss/SRqqt2/detection_loss‚0.7676624821875127 _runtime‚1313.124279648„çU é„ /loss/FRt1/assignment_loss‚2.215674236173474 1loss/FBt2/assignment_loss‚0.19906261337480372 0loss/SRqqt1/detection_loss‚1.118236096139395 /loss/FBt2/detection_loss‚0.6743463542958159 _step‚126 2loss/SRqqt1/assignment_loss‚0.7259274262215684 trainer/global_step‚1549 epoch‚3 0loss/FRt2/assignment_loss‚0.9913539078060069 1loss/FBt1/assignment_loss‚0.19879630198686624 /loss/FRt2/detection_loss‚1.6614605122690493 1loss/SRqqt2/detection_loss‚0.7676624821875127 _runtime‚1313.124279648 2loss/SRqqt2/assignment_loss‚0.6384726510191279 /loss/FRt1/detection_loss‚0.6520068438625943 /loss/FBt1/detection_loss‚1.1755424976348876 &loss/total_loss‚11.018541922971103 % _timestamp‚1.7843588210492125e+09~1~èê‚(:’ «ÏìÒüÖº½ gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯aæó¸ë‚(:­ «ÏìÒ°­´¾* disk./.usagePercent‚39.371029015934894$ disk./.usageGB‚586.6567840576172 disk.sda2.in‚0% disk.nvme0n1.out‚3847.16162109375# proc.memory.rssMB‚ 2715.75390625" disk.dm-0.out‚3847.16162109375 proc.cpu.threads‚17 disk.dm-1.out‚ 33.71484375# disk.nvme0c0n1.in‚ 2861.05859375! memory_percent‚98.47412109375 proc.memory.availableMB‚250 disk.sda3.out‚ 33.71484375 disk.drbd2165.in‚0 disk.dm-0.in‚ 2861.05859375 disk.rbd0.in‚0! disk.nvme0n1.in‚ 2861.05859375 disk.sda3.in‚0) proc.memory.percent‚16.57564640045166 cpu‚26.00613101571133 network.recv‚394902 disk.rbd1.out‚3.0625' disk.nvme0c0n1.out‚3847.16162109375 network.sent‚ 2.345248e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 33.71484375 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0£mäë›ì‚(: ºÏìÒ‘šÍº gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚284.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ó- _wandbruntime‚1352  _runtime‚1352«þ@Ù'ôÓO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14760197 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241852166401118 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241851808811868 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 $ _timestamp‚1.784358860099465e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243086576338944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1673817947230 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241410188345344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243086217960448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53657728821248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240178331414528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241410171305984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1695808623616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1352.175228578 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55353434449920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12577060 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1676387993600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243086576338944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2805438 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55353537444864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890ß…ºõØO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1676387993600 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12577058 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241410171305984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12577058 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241851808811868 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 $ _timestamp‚1.784358860099465e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 ICDeviceStatsMonitor.on_train_batch_start/oversize_ èÛÿ$!segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243086576338944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1673817947230 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241410188345344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243086217960448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53657728821248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240178331414528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241410171305984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1695808623616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1352.175228578 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55353434449920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12577060 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1676387993600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243086576338944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2805438 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55353537444864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14760197 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241852166401118 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14769263 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3421065 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243234830002688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241557429374976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3421319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55387892029952 _runtime‚1352.786503863 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55387995017728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1677400627712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12584692 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27343962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53691157579776 % _timestamp‚1.7843588607110374e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243234471617024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _step‚129 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1677059281408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2807183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1599 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243234830002688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241557412335616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6228505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53691138967552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 Büé&øãM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14769263 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3421065 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243234830002688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241557429374976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3421319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55387892029952 _runtime‚1352.786503863 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55387995017728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1677400627712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12584692 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27343962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53691157579776 % _timestamp‚1.7843588607110374e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243234471617024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _step‚129 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1677059281408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2807183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1599 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243234830002688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241557412335616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6228505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53691138967552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 B&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19946320354938507 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03810122609138489 /particle/accuracy_3_of_3‚0.9983704686164856 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8463087677955627 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17681804299354553 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6668094992637634 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.763139545917511 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *beýÀþ8jet/accuracy_1_of_3‚0.9185847043991089 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6413149833679199 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.788455605506897 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17231915891170502 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18312905728816986 +jet/accuracy_3_of_4‚0.07054692506790161 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚3 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11709098517894745 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952784776687622 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8601157665252686 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.615350067615509 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5387499928474426 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9586848020553589 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6332394480705261 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963901042938232 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7929517030715942 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28815487027168274 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6043606996536255 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.32690921425819397 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1833094358444214 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚1370.216999931 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856936931610107 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1852647066116333 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2528233826160431 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1954294890165329 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7389221787452698 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6926844716072083 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11674356460571289 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5850104093551636 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 /particle/accuracy_3_of_4‚0.9938491582870483 _step‚131 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5416997075080872 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7227099537849426 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5952953100204468 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7477113008499146 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03720971196889877 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9050390124320984 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.30441519618034363 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24708929657936096 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6698220372200012 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1950419545173645 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.605713427066803 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 (particle/accuracy‚0.8423742651939392 'particle/f_score‚0.6310509443283081 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36344632506370544 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02527041733264923 trainer/global_step‚1599 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5929202437400818 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6364323496818542 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01912364922463894 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2448294460773468 *jet/accuracy_1_of_4‚0.9695531129837036 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 /particle/accuracy_2_of_2‚0.9998962879180908 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1821022778749466 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9945229887962341 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6277950406074524 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6601634621620178 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929102659225464 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.050294287502765656 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04582523927092552 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25925394892692566 +jet/accuracy_2_of_2‚0.12770569324493408 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36430737376213074 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6819667816162109 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12999999523162842 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.35046494007110596 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.46963977813720703 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14928224682807922 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.4979773163795471 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3712519407272339 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6055464744567871 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7634381055831909 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7961073517799377 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09847014397382736 *loss/val_total_loss‚13.196963131101157 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9921076893806458 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5581841468811035 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1921062171459198 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5810539126396179 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963901042938232 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6372106671333313 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8352251648902893 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18312905728816986 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9960444569587708 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9855064153671265 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09847014397382736 +jet/accuracy_3_of_3‚0.05807662010192871 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15657250583171844 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4539872109889984 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6981104016304016 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48152613639831543 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6119996905326843 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18350479006767273 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6197916269302368 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14419305324554443 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7051592469215393 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942861199378967 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.615350067615509 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5742343068122864 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12520870566368103 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1576346606016159 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.011098379269242287 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972379803657532 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848358631134033 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7206169366836548 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5449996590614319 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24205541610717773 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1904195100069046 +particle/specificity‚0.9010514616966248 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17229020595550537 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6249207854270935 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14151525497436523 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5851786732673645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6633851528167725 *jet/accuracy_1_of_2‚0.7937932014465332 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6632278561592102 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17097771167755127 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.35926568508148193 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12611794471740723 % _timestamp‚1.7843588781409664e+09 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4863956570625305 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6090781092643738 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930506348609924 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14208891987800598 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7470647692680359 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.788455605506897 *jet/accuracy_2_of_4‚0.7544865608215332 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7861300706863403 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3174471855163574 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7982236742973328 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7411421537399292 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6597103476524353üT6ïË=€ Å{ ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9185847043991089 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6413149833679199 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.788455605506897 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17231915891170502 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18312905728816986 +jet/accuracy_3_of_4‚0.07054692506790161 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚3 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11709098517894745 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952784776687622 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8601157665252686 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.615350067615509 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5387499928474426 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9586848020553589 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6332394480705261 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963901042938232 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7929517030715942 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28815487027168274 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6043606996536255 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.32690921425819397 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1833094358444214 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚1370.216999931 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856936931610107 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1852647066116333 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2528233826160431 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1954294890165329 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7389221787452698 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6926844716072083 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11674356460571289 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5850104093551636 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 /particle/accuracy_3_of_4‚0.9938491582870483 _step‚131 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5416997075080872 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7227099537849426 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5952953100204468 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7477113008499146 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03720971196889877 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9050390124320984 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.30441519618034363 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24708929657936096 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6698220372200012 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1950419545173645 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.605713427066803 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 (particle/accuracy‚0.8423742651939392 'particle/f_score‚0.6310509443283081 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36344632506370544 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02527041733264923 trainer/global_step‚1599 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5929202437400818 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6364323496818542 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01912364922463894 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2448294460773468 *jet/accuracy_1_of_4‚0.9695531129837036 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 /particle/accuracy_2_of_2‚0.9998962879180908 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1821022778749466 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9945229887962341 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6277950406074524 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6601634621620178 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929102659225464 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.050294287502765656 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04582523927092552 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25925394892692566 +jet/accuracy_2_of_2‚0.12770569324493408 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36430737376213074 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6819667816162109 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12999999523162842 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.35046494007110596 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.46963977813720703 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14928224682807922 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.4979773163795471 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3712519407272339 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6055464744567871 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7634381055831909 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7961073517799377 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09847014397382736 *loss/val_total_loss‚13.196963131101157 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9921076893806458 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5581841468811035 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1921062171459198 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5810539126396179 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963901042938232 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6372106671333313 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8352251648902893 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18312905728816986 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9960444569587708 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9855064153671265 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09847014397382736 +jet/accuracy_3_of_3‚0.05807662010192871 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15657250583171844 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4539872109889984 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6981104016304016 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48152613639831543 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6119996905326843 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18350479006767273 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6197916269302368 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14419305324554443 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7051592469215393 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942861199378967 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.615350067615509 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5742343068122864 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12520870566368103 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1576346606016159 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.011098379269242287 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972379803657532 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848358631134033 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7206169366836548 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5449996590614319 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24205541610717773 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1904195100069046 +particle/specificity‚0.9010514616966248 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17229020595550537 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6249207854270935 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14151525497436523 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5851786732673645 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6633851528167725 *jet/accuracy_1_of_2‚0.7937932014465332 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6632278561592102 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17097771167755127 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.35926568508148193 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12611794471740723 % _timestamp‚1.7843588781409664e+09 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4863956570625305 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6090781092643738 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930506348609924 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14208891987800598 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7470647692680359 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.788455605506897 *jet/accuracy_2_of_4‚0.7544865608215332 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7861300706863403 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3174471855163574 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7982236742973328 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7411421537399292 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6597103476524353 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9832184314727783 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1411353051662445 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17017720639705658 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19946320354938507 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03810122609138489 /particle/accuracy_3_of_3‚0.9983704686164856 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8463087677955627 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17681804299354553 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6668094992637634 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.763139545917511ƒ=«j± ‚(:¦ çÏìÒ­Àô¼ disk.sda3.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 35.4453125) proc.memory.percent‚16.57567024230957 network.recv‚411577! disk.nvme0c0n1.in‚ 3007.421875 disk.sda3.out‚ 35.4453125 disk.sda1.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.out‚ 4114.23828125 disk.nvme0n1.in‚ 3007.421875 disk.sda.out‚ 35.4453125 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2715.7578125 disk.rbd1.out‚3.1875$ memory_percent‚95.46053409576416$ disk.nvme0c0n1.out‚ 4114.23828125 disk.dm-0.in‚ 3007.421875) disk./.usagePercent‚39.37110940230064$ disk./.usageGB‚586.6579818725586" disk.nvme0n1.out‚ 4114.23828125 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚26.042598286742503( proc.memory.availableMB‚ 743.74609375 network.sent‚ 2.481059e+06 disk.rbd0.out‚0xÇw‚ ‚(:’ çÏìÒ‘줽 gpu.0.gpu‚64.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j„{.‘ƒ ‚(:† öÏìÒ©Ì€» gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œjà.„ ‚(:° öÏìÒöó¡¼ network.sent‚ 2.538605e+06 disk.rbd0.out‚0$ memory_percent‚95.46167850494385( proc.memory.availableMB‚ 743.55859375$ disk./.usageGB‚586.6572341918945 disk.dm-0.in‚ 3007.421875 disk.sda.out‚ 35.74609375 disk.sda1.out‚0 cpu‚26.039178369514506 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 35.74609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚4365.39111328125 proc.cpu.threads‚17 disk.drbd2165.in‚0" proc.iŸ5memory.rssMB‚ 2715.7578125 disk.dm-1.out‚ 35.74609375 disk.rbd1.out‚3.25! disk.nvme0c0n1.in‚ 3007.421875 disk.sda1.in‚0" disk.dm-0.out‚4365.39111328125' disk.nvme0c0n1.out‚4365.39111328125 disk.nvme0n1.in‚ 3007.421875 disk.sda.in‚0 disk.sda3.in‚0) proc.memory.percent‚16.57567024230957 network.recv‚415643) disk./.usagePercent‚39.37105922482394Kãf“… ‚(:ˆ …ÐìÒ¢ «º gpu.0.gpu‚88.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ïЯ† ‚(:¤ …ÐìÒ⇨½ network.sent‚ 2.543675e+06) disk./.usagePercent‚39.37104488840201$ disk./.usageGB‚586.6570205688477 disk.nvme0n1.in‚ 3007.421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 36.10546875% proc.memory.availableMB‚ 743.53125$ memory_percent‚95.46184539794922" proc.memory.rssMB‚ 2715.7578125 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 network.recv‚419309! disk.nvme0c0n1.in‚ 3007.421875$ disk.nvme0c0n1.out‚ 4368.67578125" disk.nvme0n1.out‚ 4368.67578125 disk.sda3.in‚0 disk.dm-0.out‚ 4368.67578125 disk.rbd1.in‚0 disk.rbd1.out‚3.25 disk.sda.out‚ 36.10546875 disk.rbd0.out‚0 disk.sda2.out‚0) proc.memory.percent‚16.57567024230957 cpu‚26.035417544696568 disk.sda3.out‚ 36.10546875 disk.dm-0.in‚ 3007.421875 proc.cpu.threads‚17p!¿2‡ - _wandbruntime‚1411  _runtime‚1411‰È.îLj Á & lr-AdamW/pg1‚0.0004306557687608897 & lr-AdamW/pg2‚0.0004306557687608897 trainer/global_step‚1649 % _timestamp‚1.7843589193619962e+09 _runtime‚1411.437345533 _step‚132Ï7w·Ì‰ Æ & lr-AdamW/pg1‚0.0004306557687608897 & lr-AdamW/pg2‚0.0004306557687608897 trainer/global_step‚1649 % _timestamp‚1.7843589193619962e+09 _runtime‚1411.437345533 _step‚132„òÕNÁ2Š - _wandbruntime‚1411  _runtime‚1411+ ”ŸÚ'‹ ÔO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6448800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2895208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251154341831690 _runtime‚1411.438592077 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251154699420940 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15274999 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744475364352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚1649 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252446623180800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249412545261568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079324 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252446623180800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚250701789437952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079326 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6448560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3553352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744816703488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1765153782272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250701806477312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250701806477312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1742154159372 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252446264802304 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744475364352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚133 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250701789437952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2895211 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55345060235264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249412545261568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079324 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252446623180800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚250701789437952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079326 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6448560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3553352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744816703488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1765153782272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250701806477312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250701806477312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1742154159372 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252446264802304 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744475364352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚133 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250701789437952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2895211 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55345060235264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269…y„í2 - _wandbruntime‚1412  _runtime‚14125_Yä&Ž ÞM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13086958 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15284051 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1649 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3555452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _step‚134 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1766176249856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55378507606016 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250849030467584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1742824663204 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15294042 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745829337600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1745487991296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251301863557284 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843589201632307e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1743165213206 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2896956 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250849030467584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249559038894080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252594876844544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57144572479488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13086958 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249559055933440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745829337600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55378488993792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15284051 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1745487991296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252594876844544 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57144683855872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚1412.238763873 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55378507606016 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250849030467584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1742824663204 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15294042 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745829337600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1745487991296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251301863557284 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843589201632307e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1743165213206 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2896956 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250849030467584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249559038894080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252594876844544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57144572479488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13086958 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249559055933440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745829337600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55378488993792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15284051 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1745487991296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252594876844544 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57144683855872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚1412.238763873 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13086958 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15284051 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1649 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3555452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _step‚134 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591†† w 2 - _wandbruntime‚1412  _runtime‚1412÷Üœ®‘ ÿ 2loss/SRqqt1/assignment_loss‚0.6645561361275978 /loss/FBt2/detection_loss‚0.8976663176731635 _runtime‚ 1412.24118823 &loss/total_loss‚10.613957280198953 epoch‚4 0loss/FRt1/assignment_loss‚2.1592009976534943 0loss/FRt2/assignment_loss‚0.9464152569197258 /loss/FRt1/detection_loss‚0.6023564064200474 0loss/SRqqt1/detection_loss‚1.123884737205926 1loss/SRqqt2/detection_loss‚0.6714292926002019 2loss/SRqqt2/assignment_loss‚0.5973388591684327 /loss/FBt1/detection_loss‚1.1551462742349794 trainer/global_step‚1649 $ _timestamp‚1.784358920166147e+09 _step‚135 1loss/FBt1/assignment_loss‚0.05473660728113182 1loss/FBt2/assignment_loss‚0.08644360479029187 /loss/FRt2/detection_loss‚1.6547827901239596jÜKŒ ’ „ 2loss/SRqqt2/assignment_loss‚0.5973388591684327 /loss/FBt1/detection_loss‚1.1551462742349794 trainer/global_step‚1649 $ _timestamp‚1.784358920166147e+09 _step‚135 1loss/FBt1/assignment_loss‚0.05473660728113182 1loss/FBt2/assignment_loss‚0.08644360479029187 /loss/FRt2/detection_loss‚1.6547827901239596 2loss/SRqqt1/assignment_loss‚0.6645561361275978 /loss/FBt2/detection_loss‚0.8976663176731635 _runtime‚ 1412.24118823 &loss/total_loss‚10.613957280198953 epoch‚4 0loss/FRt1/assignment_loss‚2.1592009976534943 0loss/FRt2/assignment_loss‚0.9464152569197258 /loss/FRt1/detection_loss‚0.6023564064200474 0loss/SRqqt1/detection_loss‚1.123884737205926 1loss/SRqqt2/detection_loss‚0.6714292926002019‡÷ãpÇ““ ‚(:ˆ ”ÐìÒÉ͵¼ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë£„úÀ” ‚(:µ ”ÐìÒ§Ô¬¾' disk.nvme0c0n1.out‚4374.21630859375" disk.dm-0.out‚4374.21630859375 disk.dm-1.out‚ 36.69921875 proc.cpu.threads‚17) disk./.usagePercent‚39.37103797619859 disk.sda3.out‚ 36.69921875 disk.dm-0.in‚ 3230.3359375 disk.dm-1.in‚0 disk.rbd1.out‚3.3125 disk.rbd0.out‚0 disk.drbd2165.in‚0$ memory_percent‚95.46196460723877" proc.memory.rssMB‚ 2715.7578125" disk.nvme0c0n1.in‚ 3230.3359375 disk.sda.in‚0 disk.sda.out‚ 36.69921875 disk.sda3.in‚0 network.sent‚ 2.548588e+06 network.recv‚422661 disk.sda2.in‚0( proc.memory.availableMB‚ 743.51171875) proc.memory.percent‚16.57567024230957 disk.nvme0n1.in‚ 3230.3359375% disk.nvme0n1.out‚4374.21630859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚26.032079692839638$ disk./.usageGB‚586.6569175720215 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0›$Ô• ‚(:’ £ÐìÒ¦óÖ½ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~°V Á– ‚(:¶ £ÐìÒ÷ˆÇ¿( proc.memory.availableMB‚ 743.75390625 disk.sda.out‚ 37.08984375 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3230.3359375) proc.memory.percent‚16.57567024230957 network.recv‚427483 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 3230.3359375 network.sent‚ 2.611382e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 37.08984375" disk.dm-0.out‚4378.55029296875" proc.memory.rssMB‚ 2715.7578125 proc.cpu.threads‚17% disk.nvme0n1.out‚4378.55029296875 disk.sda1.in‚0 disk.rbd1.out‚3.3125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚4378.55029296875* disk./.usagePercent‚39.371034392093115$ disk./.usageGB‚586.6568641662598 disk.nvme0n1.in‚ 3230.3359375 disk.drbd2165.in‚0 cpu‚26.028803670079668 disk.sda3.out‚ 37.08984375 disk.dm-1.in‚0$ memory_percent‚95.46048641204834²<(Q2— - _wandbruntime‚1451  _runtime‚1451Û òȘ  _runtime‚1451.334476655 _step‚136 ' lr-AdamW/pg1‚0.00043030229073242747 ' lr-AdamW/pg2‚0.00043030229073242747 trainer/global_step‚1699 $ _timestamp‚1.784358959259179e+09£¤ÕÍ™ Ç ' lr-AdamW/pg2‚0.00043030229073242747 trainer/global_step‚1699 $ _timestamp‚1.784358959259179e+09 _runtime‚1451.334476655 _step‚136 ' lr-AdamW/pg1‚0.00043030229073242747ˆ0Èd/2š - _wandbruntime‚1451  _runtime‚1451iG‚Ú'› ÔO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259859011785728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258063857958912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1792770646280 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258063840919552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258063857958912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6640789 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259859370164224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2982458 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259859370164224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58832910766592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258530491912198 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258530849501448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6640534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15739428º9~Áœ ÙO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1795170866176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚137 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1816305415680 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2982458 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259859370164224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58832910766592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258530491912198 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258530849501448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6640534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15739428 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259859011785728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258063857958912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_starÙæ„Ø t/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1792770646280 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258063840919552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258063857958912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6640789 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259859370164224 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15738503 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 % _timestamp‚1.7843589598723946e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007623827968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258678013637792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13468558 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817419880448 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1817337602048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1796183493120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1451.947933471 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13468558 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15748494 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2984206 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58867365412864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256884589527040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15738503 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258211098988544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258211098988544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1699 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1796524839424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438]±×)é&Ÿ ãM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29207059 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57049945532416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1796524839424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1793781700114 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 % _timestamp‚1.7843589598723946e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007623827968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258678013637792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13468558 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817419880448 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1817337602048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1796183493120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1451.947933471 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13468558 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15748494 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2984206 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58867365412864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256884589527040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15738503 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258211098988544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258211098988544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1699 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1796524839424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29207059 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468556 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258211081949184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6644598 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58867264522240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3660136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260007265442304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1793441150112 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1796183493120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3660392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2984203 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15738503 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1846207707136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843387133188 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58687954700288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚1491.235506665 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267272117147648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267271758769152 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264063595409408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13842526 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265425892401152 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267272117147648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13842524 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚141 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6831525 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16183903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3761814 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1845866368000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3761551 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265425892401152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264063612448768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265906999581956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3069708 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13842524 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16193880 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265425909440512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265906641992706 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264063595409408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13842526 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265425892401152 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267272117147648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13842524 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚141 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6831525 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16183903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3761814 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1845866368000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3761551 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265425892401152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264063612448768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265906999581956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3069708 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13842524 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16193880 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265425909440512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265906641992706 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1846207707136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843387133188 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58687954700288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚1491.235506665 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267272117147648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267271758769152 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267420370811392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13850156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16202946 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16192955 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1491.848124309 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1846878994944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚1749 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1868593580032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6835376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265573150470144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266054521307662 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16192955 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58721364846592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13850158 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16202946 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58721383458816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60589958426624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1847220341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 _step‚142 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265573133430784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60590059317248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266054163718300 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844398187022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1847220341248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267420370811392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13850156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16202946 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16192955 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1491.848124309 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1846878994944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚1749 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1868593580032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6835376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265573150470144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266054521307662 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16192955 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58721364846592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13850158 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16202946 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58721383458816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60589958426624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1847220341248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 _step‚142 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.largeL·©ú]_pool.freed‚265573133430784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60590059317248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266054163718300 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844398187022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1847220341248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272787943882752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7021943 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60359374014464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14224126 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7021690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62278279615488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918800508928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚1799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271389129003008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14224124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272787943882752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1918886988800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14224124 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7021943 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60359374014464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14224126 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7021690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62278279615488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918800508928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚1799 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271389129003008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14224124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272787943882752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1918886988800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14224124 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272787943882752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231758 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274833117794816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16647407 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30889156 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843590394798357e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272935201951744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14231756 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7025670 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1897574496768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273430671388170 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231756 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272935184912384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1897915843072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272935184912384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895014673930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1897574496768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1897915843072 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3158703 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60392821385216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0­Mð¾è&à âM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3158703 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60392821385216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚146 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231758 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274833117794816 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16647407 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30889156 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843590394798357e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272935201951744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14231756 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7025670 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1897574496768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273430671388170 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231756 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272935184912384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1897915843072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272935184912384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895014673930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1897574496768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1897915843072 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3’;@ï:2Ä - _wandbruntime‚1531  _runtime‚1531{MŠë Å ƒ 2loss/FBt2/assignment_loss‚0.046952752008311796 2loss/FBt1/assignment_loss‚0.023232107977282094 /loss/FRt1/detection_loss‚0.5533328594094208 /loss/FBt1/detection_loss‚1.1682273999143105 %loss/total_loss‚9.916690624844495 0loss/FRt1/assignment_loss‚2.1224965376019136 0loss/FRt2/assignment_loss‚0.8455761254142603 2loss/SRqqt2/assignment_loss‚0.6374468677254844 /loss/FRt2/detection_loss‚1.6601446516753942 1loss/SRqqt1/detection_loss‚1.0262371042749554 epoch‚4 trainer/global_step‚1799 _runtime‚1531.556820329 1loss/SRqqt2/detection_loss‚0.6639302999224166 0loss/FBt2/detection_loss‚0.49139158519589016 $ _timestamp‚1.784359039481752e+09 _step‚147 2loss/SRqqt1/assignment_loss‚0.67772233372485664ÕÎÆ ˆ 2loss/SRqqt1/assignment_loss‚0.6777223337248566 2loss/FBt2/assignment_loss‚0.046952752008311796 2loss/FBt1/assignment_loss‚0.023232107977282094 /loss/FRt1/detection_loss‚0.5533328594094208 /loss/FBt1/detection_loss‚1.1682273999143105 %loss/total_loss‚9.916690624844495 0loss/FRt1/assignment_loss‚2.1224965376019136 0loss/FRt2/assignment_loss‚0.8455761254142603 2loss/SRqqt2/assignment_loss‚0.6374468677254844 /loss/FRt2/detection_loss‚1.6601446516753942 1loss/SRqqt1/detection_loss‚1.0262371042749554 epoch‚4 trainer/global_step‚1799 _runtime‚1531.556820329 1loss/SRqqt2/detection_loss‚0.6639302999224166 0loss/FBt2/detection_loss‚0.49139158519589016 $ _timestamp‚1.784359039481752e+09 _step‚147“ºŠ`“Ç ‚(:ˆ ŒÑìÒ¼ç¼ gpu.0.gpu‚98.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ · ÄÈ ‚(:¹ ŒÑìÒìóѼ% disk.nvme0n1.out‚4412.93505859375 disk.sda3.out‚ 40.20703125 proc.cpu.threads‚17 disk.rbd1.in‚0' disk.nvme0c0n1.out‚4412.93505859375 disk.sda.in‚0) disk./.usagePercent‚39.37105410467325( proc.memory.availableMB‚ 743.70703125 disk.sda1.out‚0 disk.dm-1.out‚ 40.20703125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 3509.1953125" disk.dm-0.out‚4412.93505859375 disk.rbd1.out‚ 3.64453125" disk.nvme0c0n1.in‚ 3509.1953125 disk.nvme0n1.in‚ 3509.1953125$ memory_percent‚95.46077251434326 cpu‚26.007861580679478 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 40.20703125 network.recv‚453418 disk.sda2.out‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2715.7578125$ disk./.usageGB‚586.6571578979492 disk.sda1.in‚0 disk.dm-1.in‚0) proc.memory.percent‚16.57567024230957 network.sent‚ 2.761974e+06 disk.drbd2165.out‚0Uʃ“É ‚(:ˆ ›ÑìÒл gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚205.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.1028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þà¾¬Ê ‚(:¡ ›ÑìÒÚÃȼ disk.nvme0n1.in‚ 3509.1953125 disk.rbd1.out‚ 3.64453125 disk.rbd1.in‚0$ memory_percent‚95.46349048614502& disk.nvme0c0n1.out‚4415.7509765625" proc.memory.rssMB‚ 2715.7578125 network.sent‚ 2.824842e+06" disk.nvme0c0n1.in‚ 3509.1953125 disk.dm-1.in‚0) proc.memory.percent‚16.57567024230957$ disk.nvme0n1.out‚4415.7509765625 disk.sda1.in‚0( proc.memory.availableMB‚ 743.26171875 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚40.5 network.recv‚458113) disk./.usagePercent‚39.37103925623626$ disk./.usageGB‚586.6569366455078 disk.rbd0.out‚0 disk.sda.out‚40.5 disk.sda3.in‚0 disk.sda3.out‚40.5 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 3509.1953125 cpu‚26.005264797885932 proc.cpu.threads‚17! disk.dm-0.out‚4415.7509765625±&—å2Ë - _wandbruntime‚1570  _runtime‚1570Ó$9XÉÌ Ã _runtime‚1570.827369593 _step‚148 ' lr-AdamW/pg1‚0.00042915994444374215 ' lr-AdamW/pg2‚0.00042915994444374215 trainer/global_step‚1849 % _timestamp‚1.7843590787520921e+09ØNc³ÎÍ È ' lr-AdamW/pg1‚0.00042915994444374215 ' lr-AdamW/pg2‚0.00042915994444374215 trainer/global_step‚1849 % _timestamp‚1.7843590787520921e+09 _runtime‚1570.827369593 _step‚148”-¿<:2Î - _wandbruntime‚1570  _runtime‚1570¼™[pÚ'Ï ÔO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14605724 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17092807 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969948624384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843590787531343e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17102784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970037201408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14605726 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14605724 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3244208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17092807 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚1849 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282097611114496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280150012403712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7212906 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62030830553088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278714679635968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3968419 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1947598710784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280150012403712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280149995364352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280150012403712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7212906 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62030830553088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278714679635968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3968419 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1947598710784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280150012403712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280149995364352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17102784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970037201408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14605726 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14605724 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3244208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0•eÄn 2Ñ - _wandbruntime‚1571  _runtime‚1571)/¿ã&Ò ÝM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278861173268480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1948269998592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14613358 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚1849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14613356 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14613356 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7216682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 $ _timestamp‚1.784359079364083e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971057270784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280297253433344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282245864778240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278861190307840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 _runtime‚1571.439796615 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚150 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280297253433344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1948611344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970968700928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17111850 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280806821468678 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3970476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280297236393984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1948269998592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1945631160838 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1948611344896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888?X®Çè&Ó âM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1945631160838 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1948611344896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278861173268480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1948269998592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14613358 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚1849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14613356 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14613356 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7216682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 $ _timestamp‚1.784359079364083e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971057270784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280297253433344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282245864778240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278861190307840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 _runtime‚1571.439796615 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚150 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280297253433344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1948611344896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970968700928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17111850 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280806821468678 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3970476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280297236393984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1948269998592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0–ù]ÅÔ2Ô - _wandbruntime‚1571  _runtime‚1571 à¨Õ ÿ trainer/global_step‚1849 % _timestamp‚1.7843590793670342e+09 _step‚151 /loss/FRt2/detection_loss‚1.6572579495651572 1loss/SRqqt1/detection_loss‚1.0961512053945979 &loss/total_loss‚10.394231798427082 0loss/FRt2/assignment_loss‚0.9224319347009367 2loss/SRqqt1/assignment_loss‚0.8193103281184307 1loss/SRqqt2/assignment_loss‚0.575756691865277 0loss/FBt2/detection_loss‚0.48102032003301354 /loss/FRt1/assignment_loss‚2.153580506700082 1loss/FBt1/assignment_loss‚0.20615153615472082 epoch‚4 _runtime‚1571.442094433 /loss/FBt2/assignment_loss‚0.084702337666442 /loss/FRt1/detection_loss‚0.5723070445789897 1loss/SRqqt2/detection_loss‚0.7594841703764434 /loss/FBt1/detection_loss‚1.0660777732729911_¸–M Ö „ /loss/FBt2/assignment_loss‚0.084702337666442 /loss/FRt1/detection_loss‚0.5723070445789897 1loss/SRqqt2/detection_loss‚0.7594841703764434 /loss/FBt1/detection_loss‚1.0660777732729911 trainer/global_step‚1849 % _timestamp‚1.7843590793670342e+09 _step‚151 /loss/FRt2/detection_loss‚1.6572579495651572 1loss/SRqqt1/detection_loss‚1.0961512053945979 &loss/total_loss‚10.394231798427082 0loss/FRt2/assignment_loss‚0.9224319347009367 2loss/SRqqt1/assignment_loss‚0.8193103281184307 1loss/SRqqt2/assignment_loss‚0.575756691865277 0loss/FBt2/detection_loss‚0.48102032003301354 /loss/FRt1/assignment_loss‚2.153580506700082 1loss/FBt1/assignment_loss‚0.20615153615472082 epoch‚4 _runtime‚1571.442094433—Àþ^:“× ‚(:ˆ ªÑìÒ—‹»¼ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åœè¶Ø ‚(:² ªÑìÒõìó½ disk.drbd2165.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2715.7578125 network.recv‚461011 disk.sda.out‚ 40.76171875 disk.sda2.in‚0 disk.dm-0.in‚ 3509.1953125 network.sent‚ 2.829776e+06 disk.rbd1.out‚ 3.70703125# disk.nvme0n1.out‚4420.791015625 disk.sda3.out‚ 40.76171875 disk.sda2.out‚0 disk.dm-0.out‚4420.791015625 disk.rbd0.in‚0% disk.nvme0c0n1.out‚4420.791015625) proc.memory.percent‚16.57567024230957) disk./.usagePercent‚39.37104949653763®_ävŸ" disk.nvme0c0n1.in‚ 3509.1953125 disk.nvme0n1.in‚ 3509.1953125 cpu‚26.00251329410051 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 743.28515625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚95.46334743499756$ disk./.usageGB‚586.6570892333984 disk.drbd2165.in‚0 disk.dm-1.out‚ 40.76171875v’‘4“Ù ‚(:ˆ ¹ÑìÒ¯Ö­» gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠëÖ»Ú ‚(:° ¹ÑìÒêùë»$ disk./.usageGB‚586.6569213867188 disk.drbd2165.out‚0$ memory_percent‚95.46449184417725 proc.cpu.threads‚17* disk./.usagePercent‚39.371038232206125 disk.rbd0.in‚0$ disk.nvme0n1.out‚4423.7587890625 disk.sda3.in‚0 network.recv‚465726 disk.nvme0n1.in‚ 3509.1953125) proc.memory.percent‚16.57567024230957 disk.sda2.in‚0 disk.dm-1.out‚ 41.140625& disk.nvme0c0n1.out‚4423.7587890625 cpu‚25.99992618733508 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.70703125 disk.rbd0.out‚0 disk.sda.out‚ 41.140625 disk.drbd2165.in‚0! disk.dm-0.out‚4423.7587890625" disk.nvme0c0n1.in‚ 3509.1953125 disk.sda.in‚0 network.sent‚ 2.892682e+06 disk.sda1.out‚0( proc.memory.availableMB‚ 743.09765625 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 3509.1953125 disk.sda3.out‚ 41.140625" proc.memory.rssMB‚ 2715.7578125þuK“Û ‚(:ˆ ÈÑìÒ—¢ø¼ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚224.914' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.9828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡`HÀÜ ‚(:µ ÈÑìÒ¤¯½ disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 41.44921875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3586.8359375 disk.sda.out‚ 41.44921875" disk.dm-0.out‚4428.92724609375" proc.memory.rssMB‚ 2715.7578125) disk./.usagePercent‚39.37104565642462$ disk./.usageGB‚586.6570320129395 disk.sda.in‚0 disk.dm-1.in‚0) proc.memory.percent‚16.57567024230957 cpu‚25.997114508195974 proc.cpu.threads‚17 network.sent‚ 2.8975e+06 disk.sda1.in‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 742.9609375 network.recv‚468887 disk.dm-0.in‚ 3586.8359375 disk.rbd1.out‚ 3.83203125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚4428.92724609375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3586.8359375% disk.nvme0n1.out‚4428.92724609375 disk.sda3.out‚ 41.44921875# memory_percent‚95.4653263092041}¥Cs2Ý - _wandbruntime‚1610  _runtime‚1610Ç®cÅÞ ¿ _runtime‚1610.715135422 _step‚152 % lr-AdamW/pg1‚0.000428751939802997 % lr-AdamW/pg2‚0.000428751939802997 trainer/global_step‚1899 % _timestamp‚1.7843591186395278e+09ó/4Êß Ä _step‚152 % lr-AdamW/pg1‚0.000428751939802997 % lr-AdamW/pg2‚0.000428751939802997 trainer/global_step‚1899 % _timestamp‚1.7843591186395278e+09 _runtime‚1610.715135422˜93¾2à - _wandbruntime‚1610  _runtime‚1610DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2021081696256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289510358097920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3331461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1994896044022 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32544562 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288035449823480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63702268479488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32534583 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚1899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286040196190208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17557236 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65723238792192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286040213229568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289510358097920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287512046845952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287512063885312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0$_iüß'â ÙO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2021081696256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289510358097920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3331461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1994896044022 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32544562 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288035449823480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63702268479488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32534583 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚1899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286040196190208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17557236 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65723238792192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286040213229568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289510358097920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287512046845952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287512063885312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591™OG'2ã - _wandbruntime‚1611  _runtime‚1611pØȯ!ä ÞM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14994956 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17566302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3333206 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65757688196608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286186706862080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17566302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289658611761664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287659304914944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287659304914944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073995 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288182971549186 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚1899 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63735697238016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286186723901440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287659287875584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1999306846720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14994958 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1999306846720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3333203 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚154 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚740719¼hkL]8 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287659287875584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1998965500416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14994958 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7407468 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2022098140672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888yfÐYé&å ãM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3333203 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚154 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7407198 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287659287875584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1998965500416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14994958 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7407468 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2022098140672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14994956 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17566302 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3333206 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65757688196608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286186706862080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17566302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289658611761664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287659304914944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287659304914944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073995 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288182971549186 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚1899 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63735697238016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286186723901440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287659287875584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1999306846720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14994958 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1999306846720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293365729783808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18011688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2072266800128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65373687793664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7594582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295411242314738 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65373706405888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294874098327552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2048648375296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15368924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296922746702848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15368926 trainer/global_step‚1949 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295411599903988 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296923105081344/"92ô ØO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293365729783808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18011688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2072266800128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65373687793664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7594582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295411242314738 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65373706405888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294874098327552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2048648375296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15368924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296922746702848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15368926 trainer/global_step‚1949 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295411599903988 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296923105081344 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2048648375296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18001711 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4175874 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2048989714432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67445861822464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144·¼_\2õ - _wandbruntime‚1651  _runtime‚1651VöÂÓã&ö ÝM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295021339357184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18020754 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293512240455680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18010763 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15376558 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15376556 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2050002348544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7598696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295021356396544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3420453 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67480420844032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4177980 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2046864134654 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15376558 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4178240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295021339357184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18020754 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297071000359424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2073197109760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18010763 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚65407135164416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293512257495040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚158 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2046523584652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983¬‡«`è&÷ âM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049661002240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚65407116552192 trainer/global_step‚1949 $ _timestamp‚1.784359159136906e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18020754 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297071000359424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2073197109760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18010763 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚65407135164416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293512257495040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚158 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2046523584652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295021339357184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18020754 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293512240455680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18010763 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15376558 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15376556 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2050002348544 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7598696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295021356396544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3420453 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67480420844032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4177980 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2046864134654 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15376558 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4178240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295021339357184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚1691.137442211 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7786038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ]ÏŠÓãIDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302236166848512 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69168453190656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3505958 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302787392395246 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2096469567728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15750526 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15750526 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69168566671360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15750524 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18466140 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302236149809152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18456163 % _timestamp‚1.7843591990618587e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123422339072(Þ,•ß'„ ÙO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2096129017838 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302236149809152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18456163 % _timestamp‚1.7843591990618587e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123422339072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18456163 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7786315 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4280080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚161 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15750524 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302787749984496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1999 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚1691.137442211 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7786038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302236166848512 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69168453190656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3505958 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚302787392395246 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2096469567728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15750526 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15750526 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69168566671360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15750524 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18466140 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302383407878144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18465215 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67078554478592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3507703 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100356504064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18475206 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4282418 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4282158 trainer/global_step‚1999 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚162 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15758158 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7790124 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69202898744832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304483747342848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124344266240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302935271710202 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2097140071560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302383390838784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 % _timestamp‚1.7843591996740625e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0@ü?X‡ ãM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15758156 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚1691.749592569 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3507706 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302383407878144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18465215 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67078554478592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3507703 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100356504064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18475206 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4282418 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚jd [ä4282158 trainer/global_step‚1999 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚162 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15758158 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7790124 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69202898744832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304483747342848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124344266240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302935271710202 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2097140071560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302383390838784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 % _timestamp‚1.7843591996740625e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67078573090816 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304484105728512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100697850368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¢a&¢‚2ˆ - _wandbruntime‚1691  _runtime‚1691H1幉 þ /loss/FRt1/assignment_loss‚2.114154351499092 2loss/SRqqt1/assignment_loss‚0.7162767724980399 2loss/FBt2/assignment_loss‚0.050412483173987145 1loss/SRqqt2/detection_loss‚0.7721461804661279 trainer/global_step‚1999 /loss/FRt2/assignment_loss‚0.760914324318218 _runtime‚1691.751465496 _step‚163 2loss/SRqqt2/assignment_loss‚0.5500756339926931 .loss/FRt2/detection_loss‚1.574328756782137 /loss/FBt1/detection_loss‚0.8652837439267723 epoch‚4 $ _timestamp‚1.784359199676396e+09 1loss/FBt1/assignment_loss‚0.02950497474327036 /loss/FRt1/detection_loss‚0.6111035135535439 1loss/SRqqt1/detection_loss‚0.9519427273126722 /loss/FBt2/detection_loss‚0.6037352409028838 %loss/total_loss‚9.599878703169438Ž2¦Õ Š ƒ 2loss/SRqqt2/assignment_loss‚0.5500756339926931 .loss/FRt2/detection_loss‚1.574328756782137 /loss/FBt1/detection_loss‚0.8652837439267723 epoch‚4 $ _timestamp‚1.784359199676396e+09 1loss/FBt1/assignment_loss‚0.02950497474327036 /loss/FRt1/detection_loss‚0.6111035135535439 1loss/SRqqt1/detection_loss‚0.9519427273126722 /loss/FBt2/detection_loss‚0.6037352409028838 %loss/total_loss‚9.599878703169438 /loss/FRt1/assignment_loss‚2.114154351499092 2loss/SRqqt1/assignment_loss‚0.7162767724980399 2loss/FBt2/assignment_loss‚0.050412483173987145 1loss/SRqqt2/detection_loss‚0.7721461804661279 trainer/global_step‚1999 /loss/FRt2/assignment_loss‚0.760914324318218 _runtime‚1691.751465496 _step‚163£Q0I¥‹ ‚(:’ ¢ÒìÒêË„½ gpu.0.gpu‚96.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HbáKµŒ ‚(:ª ¢ÒìÒðõ¤½ disk.sda.in‚0 disk.dm-1.out‚43.84375$ memory_percent‚96.86882495880127 proc.memory.rssMB‚2699.625) disk./.usagePercent‚39.59817067665946" disk.nvme0c0n1.in‚ 3663.9453125 disk.nvme0n1.in‚ 3663.9453125 disk.sda3.in‚0 disk.sda3.out‚43.84375 disk.sda1.in‚0 disk.dm-0.in‚ 3663.9453125 network.sent‚ 3.042303e+06$ disk.nvme0n1.out‚5830.4912109375 disk.drbd2165.in‚0 disk.dm-1.in‚0 cpu‚26.004683321235927 disk.rbd0.in‚0& disk.nvme0c0n1.out‚5830.4912109375 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 513.01171875 proc.cpu.threads‚17$ disk./.usageGB‚590.0413589477539 disk.sda.out‚43.84375 disk.rbd1.out‚ 4.01953125 network.recv‚490623 disk.rbd1.in‚0* proc.memory.percent‚16.477203369140625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0! disk.dm-0.out‚5830.4912109375+Þ ³2 - _wandbruntime‚1709  _runtime‚1709,nÊÂ=Ž ¼{ 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1827729344367981 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05037612095475197 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21675336360931396 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11412239074707031 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.674593985080719 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2338913232088089 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9228323698043823 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791865348816 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14083333313465118 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚4 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6980855464935303 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.216009259223938 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.168503075838089 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6434003114700317 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16533967852592468 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 'particle/accuracy‚0.842509925365448 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6036385893821716 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6506651639938354 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_4‚0.9698585271835327 particle/accuracy_1_of_3‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09764304757118225 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2605179250240326 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8242251873016357 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2236146181821823 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2803378105163574 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6463950276374817 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18774962425231934 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032080285251140594 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9085202813148499 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7940308451652527 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7852216362953186 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8098863959312439 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7014299631118774 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20955635607242584 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6247961521148682 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05835394561290741 +jet/accuracy_1_of_1‚0.33010777831077576 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6298394203186035 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.10168550908565521 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6796373724937439 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6396647691726685 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.656568706035614 +jet/accuracy_3_of_3‚0.04394211247563362 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30795207619667053 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.062255337834358215 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.762743353843689 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_2‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.04115220159292221 particle/accuracy_1_of_5‚1 )jet/accuracy_2_of_3‚0.470041424036026 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21675336360931396 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2343197762966156 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7274962067604065 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20277509093284607 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.36737319827079773 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9946556091308594 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3698451519012451 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06720609217882156 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3856460452079773 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8154572248458862 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38030165433883667 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1683460921049118 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19507381319999695 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 &particle/f_score‚0.621107816696167 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8091620206832886 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7596583366394043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6991223692893982 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7931405901908875 +particle/sensitivity‚0.6011439561843872 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27281877398490906 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6134539842605591 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6183266639709473 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8098863959312439 particle/accuracy_3_of_5‚1 trainer/global_step‚1999 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948099255561829 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9960067272186279 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7080697417259216 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884147644042969 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46962451934814453 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7836980223655701 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7754393815994263 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9938491582870483 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19599038362503052 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919737577438354 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.993181586265564 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7112219333648682 *jet/accuracy_1_of_2‚0.8021522760391235 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.630628228187561 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13168343901634216 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_2_of_4‚0.7288796305656433 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9875044822692871 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1146499291062355 /particle/accuracy_4_of_4‚0.9291108250617981 _step‚164 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03927282616496086 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9931292533874512 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951786994934082 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2056581825017929 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3746740520000458 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868414998054504 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22890442609786987 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48043641448020935 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6254959106445312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8126038908958435 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5599466562271118 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02595067024230957 (loss/val_total_loss‚9.56170182822775 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6014963388442993 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03664477542042732 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9952360987663269 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_3_of_3‚0.9994933605194092 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1769300401210785 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7793864011764526 particle/accuracy_2_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.062451764941215515 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7950316667556763 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7049440145492554 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9935680627822876 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9946556091308594 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5657835602760315 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 _runtime‚1709.985758042 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7964096069335938 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.776268482208252 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06192933768033981 particle/accuracy_2_of_3‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7183673977851868 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05804108455777168 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9864149689674377 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9886156916618347 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18633468449115753 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833432674408 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992197573184967 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06720609217882156 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7060629725456238 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8673388957977295 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18228498101234436 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 % _timestamp‚1.7843592179098024e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1654364913702011 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9287001490592957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6653874516487122 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6534125208854675 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22915466129779816 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21616490185260773 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2206847369670868 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_2‚0.11309298872947693 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6399990320205688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6534125208854675 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935Ôn„X" Á{ 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5657835602760315 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 _runtime‚1709.985758042 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7964096069335938 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.776268482208252 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06192933768033981 particle/accuracy_2_of_3‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7183673977851868 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05804108455777168 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9864149689674377 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9886156916618347 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18633468449115753 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833432674408 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992197573184967 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06720609217882156 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7060629725456238 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8673388957977295 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18228498101234436 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 % _timestamp‚1.7843592179098024e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1654364913702011 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9287001490592957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6653874516487122 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6534125208854675 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22915466129779816 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21616490185260773 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2206847369670868 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_2‚0.11309298872947693 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6399990320205688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6534125208854675 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1827729344367981 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05037612095475197 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21675336360931396 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11412239074707031 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.674593985080719 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2338913232088089 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9228323698043823 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791865348816 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14083333313465118 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚4 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6980855464935303 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.216009259223938 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.168503075838089 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6434003114700317 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16533967852592468 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 'particle/accuracy‚0.842509925365448 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6036385893821716 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6506651639938354 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_4‚0.9698585271835327 particle/accuracy_1_of_3‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09764304757118225 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2605179250240326 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8242251873016357 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2236146181821823 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2803378105163574 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6463950276374817 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.18774962425231934 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032080285251140594 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9085202813148499 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7940308451652527 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7852216362953186 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8098863959312439 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7014299631118774 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20955635607242584 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6247961521148682 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05835394561290741 +jet/accuracy_1_of_1‚0.33010777831077576 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6298394203186035 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.10168550908565521 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6796373724937439 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6396647691726685 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.656568706035614 +jet/accuracy_3_of_3‚0.04394211247563362 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30795207619667053 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.062255337834358215 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.762743353843689 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_2‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.04115220159292221 particle/accuracy_1_of_5‚1 )jet/accuracy_2_of_3‚0.470041424036026 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21675336360931396 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2343197762966156 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7274962067604065 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20277509093284607 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.36737319827079773 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9946556091308594 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3698451519012451 =&Purity/1FRt*SRóuoqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06720609217882156 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3856460452079773 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8154572248458862 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38030165433883667 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1683460921049118 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19507381319999695 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 &particle/f_score‚0.621107816696167 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8091620206832886 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7596583366394043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6991223692893982 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7931405901908875 +particle/sensitivity‚0.6011439561843872 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27281877398490906 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6134539842605591 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6183266639709473 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8098863959312439 particle/accuracy_3_of_5‚1 trainer/global_step‚1999 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948099255561829 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9960067272186279 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7080697417259216 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884147644042969 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46962451934814453 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7836980223655701 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7754393815994263 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9938491582870483 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19599038362503052 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919737577438354 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.993181586265564 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7112219333648682 *jet/accuracy_1_of_2‚0.8021522760391235 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.630628228187561 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13168343901634216 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_2_of_4‚0.7288796305656433 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9875044822692871 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1146499291062355 /particle/accuracy_4_of_4‚0.9291108250617981 _step‚164 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03927282616496086 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9931292533874512 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951786994934082 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2056581825017929 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3746740520000458 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868414998054504 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22890442609786987 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48043641448020935 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6254959106445312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8126038908958435 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5599466562271118 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02595067024230957 (loss/val_total_loss‚9.56170182822775 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6014963388442993 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03664477542042732 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9952360987663269 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_3_of_3‚0.9994933605194092 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1769300401210785 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7793864011764526 particle/accuracy_2_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.062451764941215515 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7950316667556763 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7049440145492554 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9935680627822876 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9946556091308594¤ç"ûœ ‚(:‘ ±ÒìÒ£úŠ½ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†‡ºþµ‘ ‚(:ª ±ÒìÒ몖¾) disk./.usagePercent‚39.55159599397772 disk.sda.out‚44.46875 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚495272 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.out‚44.46875 disk.rbd1.in‚0 proc.memory.rssMB‚2699.625 cpu‚26.038633338698453 disk.sda3.out‚44.46875% proc.memory.availableMB‚ 513.09375$ disk./.usageGB‚589.3473625183105 disk.nvme0n1.in‚ 3663.9453125% disk.nvme0n1.out‚5961.32958984375 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.dm-0.out‚5961.32958984375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚96.86832427978516* proc.memory.percent‚16.477203369140625 proc.cpu.threads‚17 network.sent‚ 3.105228e+06" disk.nvme0c0n1.in‚ 3663.9453125' disk.nvme0c0n1.out‚5961.32958984375 disk.drbd2165.in‚0 disk.dm-0.in‚ 3663.9453125 disk.rbd1.out‚ 4.01953125âH“’ ‚(:ˆ ÀÒìÒ¤¼Ýº gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªü_f¾“ ‚(:³ ÀÒìÒì­ê¼) proc.memory.percent‚16.57567024230957 network.sent‚ 3.162424e+06 disk.sda1.in‚0" disk.dm-0.out‚7929.60986328125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 3668.328125" proc.memory.rssMB‚ 2715.7578125 disk.sda1.out‚0 disk.dm-0.in‚ 3668.328125 disk.rbd1.out‚ 4.08203125! disk.nvme0c0n1.in‚ 3668.328125% disk.nvme0n1.out‚7929.60986328125 disk.sda.in‚0$ disk./.usageGB‚588.7338447570801 disk.sda3.out‚ 44.9765625 disk.dm-1.out‚ 44.9765625 disk.sda.out‚ 44.9765625' disk.nvme0c0n1.out‚7929.60986328125$ memory_percent‚95.47922611236572( proc.memory.availableMB‚ 740.68359375 cpu‚26.038915573155414) disk./.usagePercent‚39.51042230224585 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚499289 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚03ÄyÙ“” ‚(:ˆ ÏÒìÒäŸÕ» gpu.0.gpu‚57.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`–V¹• ‚(:® ÏÒìÒ‘“¿¼ disk.sda1.in‚0! disk.dm-0.out‚7933.4443359375 disk.rbd1.in‚0& disk.nvme0c0n1.out‚7933.4443359375 cpu‚26.041995975081473 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚95.47834396362305 disk.sda2.out‚0$ disk.nvme0n1.out‚7933.4443359375 disk.sda3.out‚ 45.265625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 45.265625) proc.memory.percent‚16.57567024230957 proc.cpu.threads‚17) disk./.usagePercent‚39.51041871814037$ disk./.usageGB‚588.7337913513184 disk.rbd1.out‚ 4.08203125" disk.nvme0c0n1.in‚ 3713.9609375 disk.nvme0n1.in‚ 3713.9609375" proc.memory.rssMB‚ 2715.7578125 network.recv‚502716 disk.dm-0.in‚ 3713.9609375 disk.rbd0.in‚0 network.sent‚ 3.167454e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 45.265625 disk.sda3.in‚0& proc.memory.availableMB‚ 740.828125Ü5œ£2– - _wandbruntime‚1751  _runtime‚1751*^ Ä— ¾ % lr-AdamW/pg1‚0.000427446625110689 % lr-AdamW/pg2‚0.000427446625110689 trainer/global_step‚2049 % _timestamp‚1.7843592597077985e+09 _runtime‚ 1751.78314035 _step‚165É"7¾É˜ à % lr-AdamW/pg2‚0.000427446625110689 trainer/global_step‚2049 % _timestamp‚1.7843592597077985e+09 _runtime‚ 1751.78314035 _step‚165 % lr-AdamW/pg1‚0.000427446625110689¥9™2™ - _wandbruntime‚1751  _runtime‚1751ø^…Ú'š ÔO YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70925211819520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313695540528128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18990919 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚166 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70925316911616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16252790 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2167772587008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68732475746304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312089925415068 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309925477224448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18980942 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8010210 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414742 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68732494358528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8010473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2049 _runtime‚1751.784373578 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2192736073216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2168113926144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2164805779870 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35233732 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312090283004318 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18980942 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8010210 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414742 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68732494358528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8010473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2049 _runtime‚1751.784373578 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2192736073216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2168113926144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2164805779870 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35233732 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312090283004318 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 H@Deviceb/b†$StatsMonitor.on_train_batch_end/inactive_split.all.current‚268 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18989994 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68765923117056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310071987896320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313844152570368 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18989994 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4416565 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8014306 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68765904504832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312237804730024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311675026010112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2168785213952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2169126560256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚167 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313844152570368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311675008970752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260422 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3597473 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2193754734592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35260409 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16260422 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4416830 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚1752.3972170263îíZé&ž ãM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260422 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3597473 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2193754734592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35260409 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16260422 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4416830 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚1752.397217026 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311675026010112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18989994 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68765923117056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310071987896320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313844152570368 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18989994 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4416565 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8014306 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68765904504832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312237804730024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311675026010112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2168785213952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2169126560256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚167 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313844152570368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311675008970752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16634392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8200792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3682978 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4517814 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72648063496704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321108287511552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319466075495576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚15080‹ß'­ ÙO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3682978 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4517814 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72648063496704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321108287511552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319466075495576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218809427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19445371 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70403932284928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2215081716888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321108645890048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318889819422720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70403913672704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321108645890048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218468088832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318889836462080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319466433084826 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317250993778688 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16634390 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218468088832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19435394 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8201061 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72647958404608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218809427968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16634392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8200792 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319613954810532 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16642022 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321256899553792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70437342431232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2216092770610 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2099 _runtime‚1792.867257792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321256541168128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19454437 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19444446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319613597221170 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19454437 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219822062080 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319037077491712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319037060452352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3684723 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70437361043456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3684726 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4519834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72682511460352 % _timestamp‚1.7843593007916903e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72682404278272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2216433320612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219480715776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4520099 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219822062080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2245150416896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219480715776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321256899553792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77%ˆí$° ãM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3684723 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70437361043456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3684726 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4519834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72682511460352 % _timestamp‚1.7843593007916903e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72682404278272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2216433320612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219480715776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4520099 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219822062080 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2245150416896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219480715776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321256899553792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8204825 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36096461 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319037077491712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319613954810532 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16642022 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321256899553792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70437342431232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2216092770610 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2099 _runtime‚1792.867257792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321256541168128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19454437 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19444446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319613597221170 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19454437 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219822062080 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319037077491712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ÉS!üGADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319037060452352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508«-æþ2± - _wandbruntime‚1792  _runtime‚1792B­h² € /loss/FRt1/assignment_loss‚2.079581747998248 0loss/FRt2/assignment_loss‚0.9034612859777836 1loss/SRqqt2/detection_loss‚0.6725998756335512 &loss/total_loss‚10.162560451260443 1loss/FBt2/assignment_loss‚0.07042916161101509 /loss/FRt1/detection_loss‚0.5401369210690113 1loss/SRqqt1/detection_loss‚0.9663751509084347 0loss/FBt2/detection_loss‚0.48141923373350437 % _timestamp‚1.7843593007934763e+09 2loss/SRqqt1/assignment_loss‚0.6703020260326882 epoch‚5 trainer/global_step‚2099 _step‚172 2loss/SRqqt2/assignment_loss‚0.6308860446023906 0loss/FBt1/assignment_loss‚0.1889051777010435 /loss/FRt2/detection_loss‚1.7499623852094317 /loss/FBt1/detection_loss‚1.2085014407833417 _runtime‚ 1792.86855409S´²ƒ ³ … trainer/global_step‚2099 _step‚172 2loss/SRqqt2/assignment_loss‚0.6308860446023906 0loss/FBt1/assignment_loss‚0.1889051777010435 /loss/FRt2/detection_loss‚1.7499623852094317 /loss/FBt1/detection_loss‚1.2085014407833417 _runtime‚ 1792.86855409 /loss/FRt1/assignment_loss‚2.079581747998248 0loss/FRt2/assignment_loss‚0.9034612859777836 1loss/SRqqt2/detection_loss‚0.6725998756335512 &loss/total_loss‚10.162560451260443 1loss/FBt2/assignment_loss‚0.07042916161101509 /loss/FRt1/detection_loss‚0.5401369210690113 1loss/SRqqt1/detection_loss‚0.9663751509084347 0loss/FBt2/detection_loss‚0.48141923373350437 % _timestamp‚1.7843593007934763e+09 2loss/SRqqt1/assignment_loss‚0.6703020260326882 epoch‚5¬eÜOrž´ ‚(:“ ‹ÓìÒ±¦Þ» gpu.0.gpu‚37.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó«´_³µ ‚(:¨ ‹ÓìÒÙöì½) disk./.usagePercent‚39.51042947045681 disk.nvme0n1.in‚ 3815.7421875 disk.dm-0.in‚ 3815.7421875( proc.memory.availableMB‚ 740.55859375 disk.rbd1.in‚0& disk.nvme0c0n1.out‚7950.8916015625 disk.dm-1.out‚46.875 disk.rbd1.out‚ 4.26953125 disk.rbd0.out‚0 disk.sda.out‚46.875 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0$ memory_percent‚95.47998905181885 network.sent‚ 3.245079e+06 disk.sda.in‚0 disk.sda3.out‚46.875 proc.cpu.threads‚17 disk.sda2.in‚0# proc.memory.rssMB‚ 2715.76171875) proc.memory.percent‚16.57569408416748 network.recv‚517681$ disk./.usageGB‚588.7339515686035 disk.sda2.out‚0 disk.drbd2165.out‚0 cpu‚26.053455605396728 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 3815.7421875$ disk.nvme0n1.out‚7950.8916015625 disk.sda1.out‚0! disk.dm-0.out‚7950.8916015625Á„¾¶ ‚(:’ šÓìÒ®Ôܼ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚231.401' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚46.28020000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ð&d÷ ‚(:¸ šÓìÒâŠú¼) disk./.usagePercent‚39.51045583923284 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚7955.87158203125# proc.memory.rssMB‚ 2715.76171875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 47.31640625 disk.sda3.in‚0 disk.sda3.out‚ 47.31640625% disk.nvme0n1.out‚7955.87158203125$ disk./.usageGB‚588.7343444824219 disk.rbd1.out‚ 4.33203125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 3815.7421875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚7955.87158203125 disk.nvme0n1.in‚ 3815.7421875 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 3.30794e+06 network.recv‚521961 disk.dm-1.out‚ 47.31640625' proc.memory.availableMB‚ 740.8046875) proc.memory.percent‚16.57569408416748 cpu‚26.056172370400816 disk.rbd1.in‚0 disk.dm-0.in‚ 3815.7421875$ memory_percent‚95.47848701477051 disk.sda1.in‚0 disk.sda2.in‚0_6Ÿö»¸ ‚(:° ©ÓìÒ½éò» disk.dm-1.out‚47.90625 proc.cpu.threads‚17 network.recv‚525680) disk./.usagePercent‚39.51043203053216" disk.nvme0c0n1.in‚ 3815.7421875' disk.nvme0c0n1.out‚7959.19970703125 disk.sda.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.57569408416748 disk.rbd1.in‚0 disk.rbd1.out‚ 4.33203125' proc.memory.availableMB‚ 740.3046875 network.sent‚ 3.312865e+06 disk.sda2.out‚0% disk.nvme0n1.out‚7959.19970703125 disk.sda3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2715.76171875 disk.sda.out‚47.90625 disk.dm-1.in‚0 cpu‚26.058445402277247 disk.rbd0.out‚0 disk.sda3.out‚47.90625 disk.drbd2165.out‚0 disk.dm-0.in‚ 3815.7421875$ disk./.usageGB‚588.7339897155762 disk.nvme0n1.in‚ 3815.7421875" disk.dm-0.out‚7959.19970703125$ memory_percent‚95.48153877258301 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 $IŸ¹ ‚(:’ ©ÓìÒš×¼ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»±‡ù2º - _wandbruntime‚1832  _runtime‚1832k#Ԙɻ à trainer/global_step‚2149 % _timestamp‚1.7843593404217865e+09 _runtime‚1832.497149263 _step‚173 ' lr-AdamW/pg1‚0.00042650894630331514 ' lr-AdamW/pg2‚0.00042650894630331514CPñ’μ È ' lr-AdamW/pg1‚0.00042650894630331514 ' lr-AdamW/pg2‚0.00042650894630331514 trainer/global_step‚2149 % _timestamp‚1.7843593404217865e+09 _runtime‚1832.497149263 _step‚173­@píã2½ - _wandbruntime‚1832  _runtime‚1832’ÇN?Ù'¾ ÓO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚326251887943680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324576527372288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19899823 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2269163590656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3770231 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266038753686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521392873472 % _timestamp‚1.7843593404229875e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015990 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72075370211328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3770228 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326842225576084 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328521034494976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2295129360384 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19889846 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2269504929792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19899823 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2295228423168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚326251870904320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74370480959488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2149 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚174 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328521034494976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2295129360384 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19889846 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2269504929792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19899823 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2295228423168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚326251870904320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74370480959488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2149 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚174 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72108780357632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2266709257518 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74404928700928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843593410369196e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚175 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72108798969856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267049807520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17023624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17023622 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19908889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326399128973312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17023624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3771973 ?6DeviceStatsMonitor.on_traÐ2‡Žin_batch_end/num_device_alloc‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4622988 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2296148343296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36922520 trainer/global_step‚2149 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3771976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326399111933952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚1833.112499884 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2270176217600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚09MH¾é& ãM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17023622 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17023624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3771973 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4622988 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2296148343296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36922520 trainer/global_step‚2149 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3771976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326399111933952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚1833.112499884 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2270176217600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚326399111933952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328669646537216 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8395228 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72108780357632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2266709257518 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74404928700928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843593410369196e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚175 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72108798969856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267049807520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17023624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17023622 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19908889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326399128973312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17397590 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2319859092480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20344298 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3857478 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843593808757792e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17397590 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335934139856896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8580987 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2320200431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17397592 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8581254 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335934139856896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73746789525504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2316314690704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333613922385920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333613939425280 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319859092480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 _runtime‚ 1872.95126743 _step‚178 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334218733245842 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20354275 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335933781478400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚331902078005248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73746808137728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0p¨<Ùl Ï ØO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2320200431616 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76093016368640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17397590 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2319859092480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20344298 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3857478 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843593808757792e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17397590 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335934139856896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8580987 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2320200431616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17397592 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8581254 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335934139856896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73746789525504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2316314690704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333613922385920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333613939425280 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319859092480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 _runtime‚ 1872.95126743 _step‚178 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334218733245842 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20354275 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335933781478400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚331902078005248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73746808137728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333613922385920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0²!‹62Ð - _wandbruntime‚1873  _runtime‚1873OŽåÕä&Ñ ÞM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336082393520640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20363341 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20353350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4725819 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1873.561727107 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17405224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73780218284032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843593814861863e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17405224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320871719424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3859226 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333761180454912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17405222 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8585045 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4725548 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336082035134976 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚179 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321213065728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3859223 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336082393520640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20353350 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333761180454912 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76127348601856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2317666294428 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2317325744426 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17405222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334365897382186 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2347225179136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚179 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321213065728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3859223 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336082393520640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20353350 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333761180454912 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76127348601856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2317666294428 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2317325744426 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17405222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334365897382186 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2347225179136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20363341 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20353350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4725819 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1873.561727107 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17405224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73780218284032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843593814861863e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17405224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320871719424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3859226 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333761180454912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17405222³Uü¯V2Ó - _wandbruntime‚1873  _runtime‚1873 šTÔ  2loss/SRqqt1/assignment_loss‚0.7395461473724849 2loss/SRqqt2/assignment_loss‚0.6003336561691628 /loss/FRt2/detection_loss‚1.5389839711750664 0loss/FRt1/assignment_loss‚1.9634322472387165 1loss/FBt2/assignment_loss‚0.03179213891356013 /loss/FBt1/detection_loss‚1.0275565444358759 epoch‚5 _runtime‚1873.564372071 _step‚180 0loss/FRt2/assignment_loss‚0.8582164349944092 1loss/SRqqt1/detection_loss‚1.0376791286917906 %loss/total_loss‚9.911096690882594 % _timestamp‚1.7843593814893317e+09 1loss/FBt1/assignment_loss‚0.15522571045767108 /loss/FRt1/detection_loss‚0.6025344776606496 1loss/SRqqt2/detection_loss‚0.7080685381650761 /loss/FBt2/detection_loss‚0.6477276956081309 trainer/global_step‚2199=qÖ Õ † 2loss/SRqqt1/assignment_loss‚0.7395461473724849 2loss/SRqqt2/assignment_loss‚0.6003336561691628 /loss/FRt2/detection_loss‚1.5389839711750664 0loss/FRt1/assignment_loss‚1.9634322472387165 1loss/FBt2/assignment_loss‚0.03179213891356013 /loss/FBt1/detection_loss‚1.0275565444358759 epoch‚5 _runtime‚1873.564372071 _step‚180 0loss/FRt2/assignment_loss‚0.8582164349944092 1loss/SRqqt1/detection_loss‚1.0376791286917906 %loss/total_loss‚9.911096690882594 % _timestamp‚1.7843593814893317e+09 1loss/FBt1/assignment_loss‚0.15522571045767108 /loss/FRt1/detection_loss‚0.6025344776606496 1loss/SRqqt2/detection_loss‚0.7080685381650761 /loss/FBt2/detection_loss‚0.6477276956081309 trainer/global_step‚2199´ïZPŽ“Ö ‚(:ˆ ÖÓìҶ賽 gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lSß»× ‚(:° ÖÓìÒ¶ãú½ proc.cpu.threads‚17 disk.rbd1.in‚0' disk.nvme0c0n1.out‚7971.79052734375 disk.nvme0n1.in‚ 3930.453125" disk.dm-0.out‚7971.79052734375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 48.8046875 disk.sda1.out‚0 disk.sda2.out‚0' disk./.usagePercent‚39.510576674789 disk.dm-1.out‚ 48.8046875 disk.sda1.in‚0 disk.dm-0.in‚ 3930.453125 cpu‚26.066346836514928 disk.sda.in‚0 disk.drbd2165.out‚0 network.recv‚536966 disk.sda.out‚ 48.8046875) proc.memory.percent‚16.57569408416748$ memory_percent‚95.47147750854492 network.sent‚ 3.385967e+06 disk.dm-1.in‚0 disk.rbd1.out‚ 4.45703125! disk.nvme0c0n1.in‚ 3930.453125$ disk./.usageGB‚588.7361450195312 disk.rbd0.in‚0% disk.nvme0n1.out‚7971.79052734375 disk.sda2.in‚0& proc.memory.availableMB‚ 741.953125# proc.memory.rssMB‚ 2715.76171875†Ñ€tÁØ ‚(:¶ åÓìÒØÜù¼# proc.memory.rssMB‚ 2715.76171875$ disk./.usageGB‚588.7344131469727 disk.rbd0.out‚0 disk.nvme0n1.in‚ 3930.453125 disk.sda1.out‚0$ memory_percent‚95.47009468078613 cpu‚26.069251120080395) disk./.usagePercent‚39.51046044736846% disk.nvme0n1.out‚7974.97412109375 disk.dm-1.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 49.00390625 proc.cpu.threads‚17 disk.sda2.out‚0) proc.memory.percent‚16.57569408416748 network.sent‚ 3.448777e+06! disk.nvme0c0n1.in‚ 3930.453125 network.recv‚541551 disk.sda.out‚ 49.00390625 disk.sda3.out‚ 49.00390625 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 3930.453125" disk.dm-0.out‚7974.97412109375 disk.rbd1.in‚0 disk.rbd1.out‚ 4.45703125' disk.nvme0c0n1.out‚7974.97412109375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 742.1796875‰ë+ŒžÙ ‚(:“ åÓìÒ³Ľ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŧó“Ú ‚(:ˆ ôÓìÒÒ¬º½ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚257.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å"^¯Û ‚(:± ôÓìÒ¥Ðɽ$ disk./.usageGB‚588.7342987060547 disk.dm-0.in‚ 3930.453125 disk.dm-1.in‚0 disk.rbd1.out‚ 4.51953125' disk.nvme0c0n1.out‚7980.56494140625 disk.sda.out‚ 49.3046875 network.sent‚ 3.453905e+06 disk.sda.in‚0( proc.memory.availableMB‚ 742.12109375! proc.memory.rssMB‚ 2715.765625) disk./.usagePercent‚39.51045276714243 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 49.3046875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 49.3046875 disk.drbd2165.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 3930.453125) proc.memory.percent‚16.57571792602539" disk.dm-0.out‚7980.56494140625% disk.nvme0n1.out‚7980.56494140625 cpu‚26.07154624010498 network.recv‚544740 disk.sda2.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 3930.453125$ memory_percent‚95.47045230865479 proc.cpuÙ¨lH .threads‚17ü¯[2Ü - _wandbruntime‚1913  _runtime‚1913æOê4ÅÝ ¿ % lr-AdamW/pg1‚0.000425517571313163 % lr-AdamW/pg2‚0.000425517571313163 trainer/global_step‚2249 % _timestamp‚1.7843594213269181e+09 _runtime‚1913.402219575 _step‚181ZOìÊÞ Ä _runtime‚1913.402219575 _step‚181 % lr-AdamW/pg1‚0.000425517571313163 % lr-AdamW/pg2‚0.000425517571313163 trainer/global_step‚2249 % _timestamp‚1.7843594213269181e+09µs‹62ß - _wandbruntime‚1913  _runtime‚1913Ë_[$Ú'à ÔO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _runtime‚1913.403471579 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341594883326350 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75418246064128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2370554594304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38587919 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339227594559488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2366931177612 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20808727 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343346886840320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4826806 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2397355779072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17779192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20808727 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3944728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343346528461824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17779192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2370895933440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2367271727502 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17779190 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2370554594304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77815601843200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340975973867520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 % _timestamp‚1.7843594213279817e+09 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339227594559488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2366931177612 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20808727 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343346886840320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4826806 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2397355779072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17779192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20808727 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3944728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343346528461824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17779192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2370895933440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2367271727502 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17779190 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2370554594304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77815601843200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340975973867520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 % _timestamp‚1.7843594213279817e+09 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2249 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20817793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341123231936512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341123214897152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843594219411154e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8775322 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75451656210432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2398286663680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2371908567552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _runtime‚1914.016645988 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4828571 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2398377330688 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20807802 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17786822 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339374105231360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2371567221248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77850052153344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341742047462694 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2367942231334 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976·}ÅLëä ãM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17786824 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339374122270720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17786822 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20807802 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3946473 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77849942874112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75451674822656 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2249 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitoÚ’±•þr.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20817793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341123231936512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341123214897152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843594219411154e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8775322 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75451656210432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2398286663680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2371908567552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _runtime‚1914.016645988 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4828571 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2398377330688 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20807802 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17786822 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339374105231360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2371567221248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚77850052153344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341742047462694 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2367942231334 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976·üïx2å - _wandbruntime‚1914  _runtime‚1914*­Z&æ € /loss/FRt1/detection_loss‚0.5434906516620494 /loss/FBt1/detection_loss‚0.9132119728019461 /loss/FBt2/detection_loss‚0.5174700824813988 trainer/global_step‚2249 /loss/FRt1/assignment_loss‚2.024356975422838 2loss/SRqqt2/assignment_loss‚0.6383830492660751 1loss/FBt1/assignment_loss‚0.22507144040825247 1loss/SRqqt2/detection_loss‚0.7331842279113264 2loss/FBt2/assignment_loss‚0.061093167680675436 /loss/FRt2/detection_loss‚1.5960914239105113 0loss/SRqqt1/detection_loss‚1.061022412936697 %loss/total_loss‚9.983628500253964 % _timestamp‚1.7843594219430442e+09 _step‚184 epoch‚5 _runtime‚1914.018121683 0loss/FRt2/assignment_loss‚0.9547163740262781 2loss/SRqqt1/assignment_loss‚0.7155367217459169×ÆRy ç … 1loss/SRqqt2/detection_loss‚0.7331842279113264 2loss/FBt2/assignment_loss‚0.061093167680675436 /loss/FRt2/detection_loss‚1.5960914239105113 0loss/SRqqt1/detection_loss‚1.061022412936697 %loss/total_loss‚9.983628500253964 % _timestamp‚1.7843594219430442e+09 _step‚184 epoch‚5 _runtime‚1914.018121683 0loss/FRt2/assignment_loss‚0.9547163740262781 2loss/SRqqt1/assignment_loss‚0.7155367217459169 /loss/FRt1/detection_loss‚0.5434906516620494 /loss/FBt1/detection_loss‚0.9132119728019461 /loss/FBt2/detection_loss‚0.5174700824813988 trainer/global_step‚2249 /loss/FRt1/assignment_loss‚2.024356975422838 2loss/SRqqt2/assignment_loss‚0.6383830492660751 1loss/FBt1/assignment_loss‚0.22507144040825247¸©ˆ4÷‘è ‚(:† ƒÔìÒÁ÷þº gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÄÆĽé ‚(:² ƒÔìÒÞÛ—¾' disk.nvme0c0n1.out‚7983.94677734375! proc.memory.rssMB‚ 2715.765625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚7983.94677734375" disk.nvme0c0n1.in‚ 4039.5234375% disk.nvme0n1.out‚7983.94677734375 disk.sda.out‚ 49.546875 disk.rbd1.out‚ 4.51953125) proc.memory.percent‚16.57571792602539 network.recv‚548448 disk.sda3.out‚ 49.546875 disk.dm-0.in‚ 4039.5234375 disk.sda3.in‚0 disk.sda1.out‚0 cpu‚26.073933877207303 network.sent‚ 3.45878e+06* disk./.usagePercent‚39.510453279157495 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4039.5234375 disk.sda2.out‚0 disk.dm-1.out‚ 49.546875( proc.memory.availableMB‚ 741.83203125$ disk./.usageGB‚588.7343063354492 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚95.47221660614014 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.in‚09Wê ‚(:’ ’ÔìÒ¡•µ» gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÉéÂë ‚(:· ’ÔìÒ©ðš¾" disk.nvme0c0n1.in‚ 4039.5234375 disk.sda3.out‚ 50.04296875( proc.memory.availableMB‚ 742.01953125) proc.memory.percent‚16.57571792602539 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 50.04296875$ memory_percent‚95.47107219696045 disk.dm-1.out‚ 50.04296875 disk.rbd1.out‚ 4.64453125' disk.nvme0c0n1.out‚7989.44677734375% disk.nvme0n1.out‚7989.44677734375 disk.sda3.in‚0! proc.memory.rssMB‚ 2715.765625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 3.521682e+06 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚39.51056925057051$ disk./.usageGB‚588.7360343933105" disk.dm-0.out‚7989.44677734375 disk.nvme0n1.in‚ 4039.5234375 disk.sda1.in‚0 cpu‚26.07591946440501 disk.dm-0.in‚ 4039.5234375 network.recv‚553258 disk.rbd1.in‚0 disk.rbd0.in‚0{ U‰¶ì ‚(:« ¡ÔìÒ۽ɼ network.sent‚ 3.526713e+06 disk.sda.out‚ 50.32421875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚26.078046432502287 disk.nvme0n1.in‚ 4039.5234375 disk.sda.in‚0 disk.sda3.out‚ 50.32421875 disk.sda2.out‚0 disk.rbd1.out‚ 4.64453125$ memory_percent‚96.97470664978027) disk./.usagePercent‚39.51029813859175 disk.sda1.in‚0 disk.dm-0.out‚ 7994.1484375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4039.5234375' proc.memory.availableMB‚ 495.6640625 network.recv‚556858$ disk./.usageGB‚588.7319946289062! disk.nvme0n1.out‚ 7994.1484375 disk.dm-0.in‚ 4039.5234375 disk.dm-1.out‚ 50.32421875 disk.rbd1.in‚0) proc.memory.percent‚16.57571792602539 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.out‚ 7994.1484375! proc.memory.rssMB‚ 2715.765625 disk.rbd0.in‚0Ô&:L“í ‚(:ˆ ¡ÔìÒŽ¹¼ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jš2î - _wandbruntime‚1953  _runtime‚1953ÃáÉï à ' lr-AdamW/pg1‚0.00042500182680386874 ' lr-AdamW/pg2‚0.00042500182680386874 trainer/global_step‚2299 % _timestamp‚1.7843594618319526e+09 _runtime‚1953.907313525 _step‚185Ÿ5¾WÎð È _runtime‚1953.907313525 _step‚185 ' lr-AdamW/pg1‚0.00042500182680386874 ' lr-AdamW/pg2‚0.00042500182680386874 trainer/global_step‚2299 % _timestamp‚1.7843594618319526e+09¹wx¹2ñ - _wandbruntime‚1953  _runtime‚1953¸q#Ù'ò ÓO _runtime‚1953.908736141 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18160790 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21263179 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348338042388480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8961578 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 $ _timestamp‚1.784359461833093e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4031981 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18160792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18160790 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4929597 trainer/global_step‚2299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348970675817608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79538063400448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18160792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348338025349120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2421591435264 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8961295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348971033406858 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2417547664520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4031978 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4929317 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21253202 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8961578 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 $ _timestamp‚1.784359461833093e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4031981 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18160792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18160790 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4929597 trainer/global_step‚2299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348970675817608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79538063400448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18160792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348338025349120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2421591435264 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8961295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348971033406858 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2417547664520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3€ˆCRY57589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4031978 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4929317 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21253202 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349118555132564 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18168424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349118197543202 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350907887487488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2422262723072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2422604069376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2449304407040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4931360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18168422 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21272245 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21272245 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4931623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚1954.521698322 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21262254 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39430676 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21262254 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4033726 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79572398543872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2418558718242 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348485266378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348485266378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2422604069376 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21272245 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21272245 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4931623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚1954.521698322 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21262254 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39430676 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21262254 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4033726 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79572398543872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2418558718242 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348485266378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348485266378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2422604069376 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349118555132564 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18168424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349118197543202 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350907887487488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2422262723072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2422604069376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2449304407040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0»YªjS2÷ - _wandbruntime‚1954  _runtime‚1954Ìú\ø  1loss/FBt1/assignment_loss‚0.06699707763770373 2loss/FBt2/assignment_loss‚0.016664613127696182 1loss/SRqqt1/detection_loss‚1.0551494649354074 /loss/FBt1/detection_loss‚1.5449941519958277 &loss/total_loss‚10.663263292740636 _runtime‚1954.523245893 /loss/FRt1/assignment_loss‚2.005172862771009 0loss/FRt2/assignment_loss‚1.0638812932432915 2loss/SRqqt1/assignment_loss‚0.7741549863146103 % _timestamp‚1.7843594624481692e+09 epoch‚5 .loss/FRt1/detection_loss‚0.563281122692322 1loss/SRqqt2/detection_loss‚0.7241089888146777 /loss/FBt2/detection_loss‚0.5012690440227743 trainer/global_step‚2299 2loss/SRqqt2/assignment_loss‚0.6235064301623288 /loss/FRt2/detection_loss‚1.7240832570229885 _step‚188ï ¼é ù † _step‚188 1loss/FBt1/assignment_loss‚0.06699707763770373 2loss/FBt2/assignment_loss‚0.016664613127696182 1loss/SRqqt1/detection_loss‚1.0551494649354074 /loss/FBt1/detection_loss‚1.5449941519958277 &loss/total_loss‚10.663263292740636 _runtime‚1954.523245893 /loss/FRt1/assignment_loss‚2.005172862771009 0loss/FRt2/assignment_loss‚1.0638812932432915 2loss/SRqqt1/assignment_loss‚0.7741549863146103 % _timestamp‚1.7843594624481692e+09 epoch‚5 .loss/FRt1/detection_loss‚0.563281122692322 1loss/SRqqt2/detection_loss‚0.7241089888146777 /loss/FBt2/detection_loss‚0.5012690440227743 trainer/global_step‚2299 2loss/SRqqt2/assignment_loss‚0.6235064301623288 /loss/FRt2/detection_loss‚1.7240832570229885¼’Ðqžú ‚(:“ °ÔìÒ¢Ú½ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âí¹û ‚(:® °Ôìҡȉ¾ proc.cpu.threads‚17$ disk./.usageGB‚588.7332229614258 disk.sda1.out‚0 disk.rbd1.out‚ 4.70703125' disk.nvme0c0n1.out‚8001.46533203125) proc.memory.percent‚16.57571792602539 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 4039.5234375' proc.memory.availableMB‚ 495.1484375" disk.dm-0.out‚8001.46533203125 network.recv‚560210) disk./.usagePercent‚39.51038057301777 disk.rbd0.out‚0 network.sent‚ 3.544622e+06 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚50.78125! proc.memory.rssMB‚ 2715.765625 disk.dm-1.out‚50.78125 disk.drbd2165.in‚0 disk.dm-0.in‚ 4039.5234375$ memory_percent‚96.97785377502441% disk.nvme0n1.out‚8001.46533203125 disk.sda.out‚50.78125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 4039.5234375 cpu‚26.079804572708547ôÖžü ‚(:“ ¿ÔìÒ„Ô¼ gpu.0.gpu‚84.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ËæÃý ‚(:¸ ¿ÔìÒŒóð¼ disk.sda3.out‚ 51.33984375( proc.memory.availableMB‚ 494.89453125) proc.memory.percent‚16.57571792602539 network.sent‚ 3.594393e+06 network.recv‚565162 disk.dm-0.in‚ 4109.5078125" disk.dm-0.out‚8005.01318359375 disk.rbd1.out‚ 4.70703125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 4109.5078125% disk.nvme0n1.out‚8005.01318359375 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4109.5078125 disk.sda.out‚ 51.33984375 cpu‚26.08223338107336* disk./.usagePercent‚39.510379036972566 disk.drbd2165.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚8005.01318359375 disk.sda2.in‚0$ disk./.usageGB‚588.7332000732422 disk.dm-1.out‚ 51.33984375 disk.rbd0.out‚0$ memory_percent‚96.97940349578857! proc.memory.rssMB‚ 2715.765625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda.in‚0•ªkæ2þ - _wandbruntime‚1994  _runtime‚1994¼ÇèÆÿ À & lr-AdamW/pg1‚0.0004244727548327953 & lr-AdamW/pg2‚0.0004244727548327953 trainer/global_step‚2349 $ _timestamp‚1.784359502306762e+09 _runtime‚1994.382079599 _step‚189ƒ“ÃË€ Å _runtime‚1994.382079599 _step‚189 & lr-AdamW/pg1‚0.0004244727548327953 & lr-AdamW/pg2‚0.0004244727548327953 trainer/global_step‚2349 $ _timestamp‚1.784359502306762e+09½]óŠ2 - _wandbruntime‚1994  _runtime‚1994ï¢a©D‚ ÒO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40250044 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21717631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78761103304704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18542390 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5032306 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81260508041728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21717631 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9151816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356346825898116 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2499386124800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4119231 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358172380807168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355700093870080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18542392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353878678786048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355700093870080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355700076830720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2468164151428 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5032585 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78761121916928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2471945597952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2472286937088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2471945597952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚2349 # _timestamp‚1.78435950230783e+09 _step‚190 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21707654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358172022428672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353878661746688 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21707654 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81260394561024 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18542392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 _runtime‚1994.383374985(|InÝ'ƒ ×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21707654 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81260394561024 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18542392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 _runtime‚1994.383374985 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40250044 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21717631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78761103304704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18542390 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5032306 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81260508041728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21717631 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9151816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356346825898116 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2499386124800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4119231 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358172380807168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355700093870080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18542392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353878678786048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355700093870080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355700076830720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2468164151428 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5032585 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78761121916928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2471945597952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2472286937088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2471945597952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚2349 # _timestamp‚1.78435950230783e+09 _step‚190 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21707654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358172022428672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353878661746688 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18550022 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355847317860352 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81294951000064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356494347623710 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2469515755152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358320634470912 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355847334899712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500400324608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354025189457920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4120976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500311754752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18550024 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78794550675456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚2349 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18550022 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21716706 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358320276085248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81294843817984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355847317860352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21726697 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2472958224896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354025172418560 _runtime‚1994.996415316 _step‚191 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18550024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356494705213072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784359502920934e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355847317860352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21726697 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2472958224896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354025172418560 _runtime‚1994.996415316 _step‚191 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18550024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356494705213072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784359502920934e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18550022 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355847317860352 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81294951000064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356494347623710 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2469515755152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358320634470912 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355847334899712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500400324608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354025189457920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4120976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500311754752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18550024 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactiÞ!‚‘ive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78794550675456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚2349 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18550022 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21716706 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358320276085248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81294843817984¿µ¸ˆ2‡ - _wandbruntime‚1994  _runtime‚1994*ôˆ  2loss/SRqqt2/assignment_loss‚0.6419523703693275 %loss/total_loss‚9.487185974954578 epoch‚5 trainer/global_step‚2349 /loss/FRt1/assignment_loss‚2.041334978785896 /loss/FBt1/detection_loss‚0.8254980974209806 _runtime‚1994.997965984 /loss/FRt2/detection_loss‚1.5797143166020475 0loss/FBt2/detection_loss‚0.44343297974543006 % _timestamp‚1.7843595029228992e+09 _step‚192 1loss/FBt1/assignment_loss‚0.04960126168740923 1loss/FBt2/assignment_loss‚0.04939621632986228 /loss/FRt1/detection_loss‚0.5888916894966304 1loss/SRqqt1/detection_loss‚1.0453937370830135 1loss/SRqqt2/detection_loss‚0.6877588973310526 0loss/FRt2/assignment_loss‚0.8387962912507447 2loss/SRqqt1/assignment_loss‚0.6954151388521825Ô4 ‰ † 2loss/SRqqt1/assignment_loss‚0.6954151388521825 2loss/SRqqt2/assignment_loss‚0.6419523703693275 %loss/total_loss‚9.487185974954578 epoch‚5 trainer/global_step‚2349 /loss/FRt1/assignment_loss‚2.041334978785896 /loss/FBt1/detection_loss‚0.8254980974209806 _runtime‚1994.997965984 /loss/FRt2/detection_loss‚1.5797143166020475 0loss/FBt2/detection_loss‚0.44343297974543006 % _timestamp‚1.7843595029228992e+09 _step‚192 1loss/FBt1/assignment_loss‚0.04960126168740923 1loss/FBt2/assignment_loss‚0.04939621632986228 /loss/FRt1/detection_loss‚0.5888916894966304 1loss/SRqqt1/detection_loss‚1.0453937370830135 1loss/SRqqt2/detection_loss‚0.6877588973310526 0loss/FRt2/assignment_loss‚0.8387962912507447À¯þVêÀŠ ‚(:µ ÎÔìÒæÖí»" disk.dm-0.out‚8010.51318359375 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚8010.51318359375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 51.72265625 disk.rbd0.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 494.671875) proc.memory.percent‚16.57571792602539 network.recv‚568498 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4109.5078125 disk.sda.out‚ 51.72265625 disk.rbd1.in‚0 disk.rbd1.out‚ 4.76953125" disk.nvme0c0n1.in‚ 4109.5078125 proc.cpu.threads‚17 network.sent‚ 3.599324e+06$ disk./.usageGB‚588.7332229614258 disk.sda3.out‚ 51.72265625$ memory_percent‚96.98076248168945 disk.sda2.in‚0% disk.nvme0n1.out‚8010.51318359375! proc.memory.rssMB‚ 2715.765625 disk.sda.in‚0 cpu‚26.08412645634386) disk./.usagePercent‚39.51038057301777 disk.sda2.out‚0 disk.dm-0.in‚ 4109.5078125Okp“‹ ‚(:ˆ ÎÔìÒñÚ ½ gpu.0.gpu‚90.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0΃îmžŒ ‚(:“ ÝÔìÒÁÏ…» gpu.0.gpu‚72.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02”+ŠÃ ‚(:¸ ÝÔìÒâ¬Þ¼# disk./.usageGB‚588.733211517334 disk.sda.out‚ 52.05078125 disk.dm-0.in‚ 4109.5078125$ memory_percent‚96.97883129119873 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚8013.32958984375% disk.nvme0n1.out‚8013.32958984375 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 494.98828125! proc.memory.rssMB‚ 2715.765625 network.sent‚ 3.662151e+06 disk.sda3.out‚ 52.05078125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 52.05078125 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4109.5078125 disk.nvme0n1.in‚ 4109.5078125 disk.sda3.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.57571792602539 cpu‚26.086554357085802 network.recv‚573508 disk.rbd1.in‚0 disk.rbd1.out‚ 4.76953125" disk.dm-0.out‚8013.32958984375* disk./.usagePercent‚39.510379804995175mÏ~ËžŽ ‚(:“ ìÔìÒçòòº gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.593' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.718599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;$ž ¸ ‚(:­ ìÔìÒ½®º½ disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.out‚8019.552734375 disk.dm-1.in‚0 cpu‚26.08870808189603$ disk./.usageGB‚588.7332191467285 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4109.5078125$ memory_percent‚96.98195457458496' proc.memory.availableMB‚ 494.4765625 network.recv‚576728 disk.sda2.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚8019.552734375 disk.sda.in‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2715.765625 disk.sda3.in‚0 disk.sda3.out‚ 52.4140625 disk.sda1.out‚0 disk.dm-1.out‚ 52.4140625 disk.rbd1.out‚ 4.83203125) proc.memory.percent‚16.57571792602539 proc.cpu.threads‚17 network.sent‚ 3.667221e+06) disk./.usagePercent‚39.51038031701024 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 4109.5078125% disk.nvme0c0n1.out‚8019.552734375 disk.sda.out‚ 52.4140625 disk.dm-0.in‚ 4109.5078125.^ô2 - _wandbruntime‚2034  _runtime‚2034½÷“hÇ‘ Á & lr-AdamW/pg1‚0.0004239303893812467 & lr-AdamW/pg2‚0.0004239303893812467 trainer/global_step‚2399 % _timestamp‚1.7843595428367958e+09 _runtime‚2034.912160679 _step‚193Ch=Ì’ Æ % _timestamp‚1.7843595428367958e+09 _runtime‚2034.912160679 _step‚193 & lr-AdamW/pg1‚0.0004239303893812467 & lr-AdamW/pg2‚0.0004239303893812467 trainer/global_step‚2399ÁZxǾ2“ - _wandbruntime‚2034  _runtime‚2034õWã Û'” ÕO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22162106 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82982758524416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2519121188226 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9341634 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126063616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22172083 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41086096 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82982884588032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363723333567874 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107451392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚194 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361204212379648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18923990 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550324744704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22162106 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363722975978624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361204195340288 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2034.913430236 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4206481 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4206478 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22172083 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363062145351680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18923990 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363062128312320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18923992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365585127790592 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2522641099776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80432559843328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2522982438912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2522982438912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365584769412096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18923992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363062145351680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2399 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363062128312320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276å}j'à'• ÚO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9341634 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126063616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22172083 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41086096 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82982884588032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363723333567874 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107451392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚194 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361204212379648 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18923990 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550324744704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22162106 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363722975978624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361204195340288 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2034.913430236 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4206481 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4206478 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22172083 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363062145351680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18923990 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363062128312320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18923992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365585127790592 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2522641099776 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80432559843328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2522982438912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2522982438912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365584769412096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18923992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363062145351680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2399 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363062128312320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22162106 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82982758524416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2519121188226Âê s2– - _wandbruntime‚2035  _runtime‚2035*š"¾— ßM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18931624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22181149 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2523995073024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2523995073024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18931624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2523653726720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22181149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83017205314560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚2399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 G:DeviceStatsMonL&ß'itor.on_train_batch_end/allocation.all.freed‚41102780 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4208223 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2551336477696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361350706012160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365733023068672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2523653726720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9345422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363209386381312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363870855293580 _step‚195 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚2399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41102780 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4208223 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2551336477696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361350706012160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365733023068672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2523653726720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9345422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363209386381312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363870855293580 _step‚195 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18931624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22181149 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2523995073024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2523995073024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18931624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773ÃóFÈ2™ - _wandbruntime‚2035  _runtime‚2035?JŒ š þ %loss/total_loss‚9.758834995134652 2loss/SRqqt1/assignment_loss‚0.6377880947675355 1loss/FBt2/assignment_loss‚0.08924481006338753 _runtime‚2035.527702698 .loss/FRt1/assignment_loss‚2.00931579981935 1loss/FBt1/assignment_loss‚0.03272124048430669 /loss/FRt1/detection_loss‚0.5526113761575966 /loss/FRt2/detection_loss‚1.5335878497033746 1loss/SRqqt1/detection_loss‚1.0501914732816695 1loss/SRqqt2/detection_loss‚0.6659898925813269 .loss/FBt1/detection_loss‚1.069760880083777 epoch‚5 0loss/FRt2/assignment_loss‚0.9432250803680782 /loss/FBt2/detection_loss‚0.5819230051041185 trainer/global_step‚2399 % _timestamp‚1.7843595434526534e+09 _step‚196 2loss/SRqqt2/assignment_loss‚0.59247549272013262òN1 › ƒ 2loss/SRqqt2/assignment_loss‚0.5924754927201326 %loss/total_loss‚9.758834995134652 2loss/SRqqt1/assignment_loss‚0.6377880947675355 1loss/FBt2/assignment_loss‚0.08924481006338753 _runtime‚2035.527702698 .loss/FRt1/assignment_loss‚2.00931579981935 1loss/FBt1/assignment_loss‚0.03272124048430669 /loss/FRt1/detection_loss‚0.5526113761575966 /loss/FRt2/detection_loss‚1.5335878497033746 1loss/SRqqt1/detection_loss‚1.0501914732816695 1loss/SRqqt2/detection_loss‚0.6659898925813269 .loss/FBt1/detection_loss‚1.069760880083777 epoch‚5 0loss/FRt2/assignment_loss‚0.9432250803680782 /loss/FBt2/detection_loss‚0.5819230051041185 trainer/global_step‚2399 % _timestamp‚1.7843595434526534e+09 _step‚196Ä¡ÐÆm‘œ ‚(:† ûÔìÒùIJ¼ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´$áØÇ ‚(:¼ ûÔìҵˋ½ disk.sda2.in‚0 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 4183.78515625 disk.sda.out‚ 52.87890625 disk.sda3.out‚ 52.87890625) disk./.usagePercent‚39.51038978928901 disk.sda2.out‚0( proc.memory.availableMB‚ 511.08984375* proc.memory.percent‚16.477251052856445 cpu‚26.09800379958555$ disk./.usageGB‚588.7333602905273% disk.nvme0n1.out‚8030.44384765625 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 4183.78515625 disk.dm-1.in‚0 disk.dm-1.out‚ 52.87890625" proc.memory.rssMB‚ 2699.6328125 network.recv‚580185' disk.nvme0c0n1.out‚8030.44384765625! disk.nvme0n1.in‚ 4183.78515625 disk.sda1.in‚0 disk.rbd1.out‚ 4.83203125 disk.rbd0.in‚0 network.sent‚ 3.672166e+06 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚96.88055515289307 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.dm-0.out‚8030.44384765625¨Ì2ž - _wandbruntime‚2053  _runtime‚2053&÷Åç5Ÿ ¡{ <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.543201744556427 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1338430792093277 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07373600453138351 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5732724070549011 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.077447310090065 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8084046244621277 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9945520758628845 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9778393507003784 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.753699779510498 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7543461918830872 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8765080571174622 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7632652521133423 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6425862908363342 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6983680129051208 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7309468984603882 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9767573475837708 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9031988978385925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23696599900722504 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934669733047485 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 _step‚197 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21196988224983215 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7212768793106079 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2186402827501297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893440008163452 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9743887782096863 particle/accuracy_5_of_5‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023983720690011978 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9746638536453247 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6914347410202026 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9873902201652527 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7015141844749451 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05947190150618553 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2146914303302765 *jet/accuracy_2_of_3‚0.5506022572517395 particle/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7557104825973511 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8743008971214294 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13560816645622253 +jet/accuracy_3_of_4‚0.09347696602344513 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833432674408 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.990214467048645 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6086717844009399 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03715694695711136 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1196034774184227 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5208333134651184 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6629298329353333 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899956583976746 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23267047107219696 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.8069490790367126 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8645880818367004 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6511443257331848 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6483690738677979 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9960067272186279 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9796942472457886 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.654110848903656 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7263944149017334 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6676191091537476 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.29629629850387573 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.758139967918396 particle/accuracy_4_of_5‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.672911524772644 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2194826453924179 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 &particle/f_score‚0.648022472858429 _runtime‚2053.718286004 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7002130746841431 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_4‚0.9697085618972778 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6964973211288452 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 particle/accuracy_3_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17696204781532288 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7647171020507812 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3305867910385132 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19057881832122803 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4774065613746643 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04758862778544426 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6284440755844116 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6350177526473999 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6940282583236694 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.654110848903656 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20250606536865234 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20325954258441925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38624483346939087 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6399670839309692 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6000674366950989 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.167484313249588 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1902417242527008 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.030325455591082573 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3885406255722046 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2827335596084595 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9795948266983032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17805978655815125 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4444444477558136 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8240740299224854 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18133677542209625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7423892617225647 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9068641662597656 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9593886137008667 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921640753746033 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.91987144947052 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 6validation_average_jet_accuracy‚0.3738383650779724 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3184310495853424 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8592198491096497 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.983917236328125 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9945520758628845 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7126028537750244 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6537398099899292 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22974207997322083 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4677712023258209 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19271919131278992 particle/accuracy_1_of_1‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_4_of_4‚0.9686647057533264 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23138584196567535 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.046995602548122406 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4480851888656616 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8645375370979309 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6400855183601379 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 epoch‚5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9920775890350342 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20873716473579407 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8685563206672668 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7691887617111206 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9944215416908264 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2194826453924179 particle/accuracy_2_of_3‚1 )loss/val_total_loss‚9.236175008933017 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28031906485557556 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9840469360351562 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9219202399253845 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7222222089767456 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7800129652023315 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7572121024131775 +jet/accuracy_3_of_3‚0.06949874013662338 (particle/accuracy‚0.8501054048538208 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28539976477622986 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0653291866183281 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 trainer/global_step‚2399 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9846875071525574 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8084046244621277 particle/accuracy_1_of_3‚1 &í¥$À Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8236206769943237 particle/accuracy_2_of_4‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0653291866183281 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6499655842781067 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.434629887342453 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.29629629850387573 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15035906434059143 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07379689812660217 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.40985026955604553 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8708428144454956 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07739543169736862 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7790617346763611 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21916064620018005 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7005074620246887 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9879844188690186 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5420531630516052 *jet/accuracy_2_of_2‚0.1450524926185608 ,jet/accuracy_4_of_4‚0.013899499550461769 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5495200753211975 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843595616422513e+09 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4444444477558136 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23157887160778046 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39856287837028503 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085­(N¬=  ¦{ ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0653291866183281 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6499655842781067 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.434629887342453 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.29629629850387573 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15035906434059143 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07379689812660217 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.40985026955604553 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8708428144454956 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07739543169736862 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7790617346763611 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21916064620018005 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7005074620246887 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9879844188690186 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5420531630516052 *jet/accuracy_2_of_2‚0.1450524926185608 ,jet/accuracy_4_of_4‚0.013899499550461769 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5495200753211975 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843595616422513e+09 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4444444477558136 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23157887160778046 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39856287837028503 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.543201744556427 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1338430792093277 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07373600453138351 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5732724070549011 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.077447310090065 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8084046244621277 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9945520758628845 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9778393507003784 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.753699779510498 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7543461918830872 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8765080571174622 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7632652521133423 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6425862908363342 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6983680129051208 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7309468984603882 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9767573475837708 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9031988978385925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23696599900722504 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934669733047485 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 _step‚197 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21196988224983215 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7212768793106079 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2186402827501297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893440008163452 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9743887782096863 particle/accuracy_5_of_5‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023983720690011978 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9746638536453247 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6914347410202026 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9873902201652527 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7015141844749451 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05947190150618553 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2146914303302765 *jet/accuracy_2_of_3‚0.5506022572517395 particle/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7557104825973511 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8743008971214294 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13560816645622253 +jet/accuracy_3_of_4‚0.09347696602344513 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833432674408 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.990214467048645 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6086717844009399 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03715694695711136 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1196034774184227 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5208333134651184 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6629298329353333 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899956583976746 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23267047107219696 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.8069490790367126 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8645880818367004 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6511443257331848 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6483690738677979 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9960067272186279 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9796942472457886 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.654110848903656 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7263944149017334 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6676191091537476 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.29629629850387573 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.758139967918396 particle/accuracy_4_of_5‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.672911524772644 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2194826453924179 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 &particle/f_score‚0.648022472858429 _runtime‚2053.718286004 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7002130746841431 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_4‚0.9697085618972778 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6964973211288452 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 particle/accuracy_3_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17696204781532288 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7647171020507812 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3305867910385132 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19057881832122803 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4774065613746643 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04758862778544426 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6284440755844116 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6350177526473999 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6940282583236694 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.654110848903656 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20250606536865234 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20325954258441925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38624483346939087 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6399670839309692 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6000674366950989 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.167484313249588 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1902417242527008 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.030325455591082573 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3885406255722046 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2827335596084595 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9795948266983032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17805978655815125 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4444444477558136 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8240740299224854 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18133677542209625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7423892617225647 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9068641662597656 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9593886137008667 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921640753746033 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.91987144947052 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 6validation_average_jet_accuracy‚0.3738383650779724 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3184310495853424 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8592198491096497 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.983917236328125 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9945520758628845 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7126028537750244 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6537398099899292 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22974207997322083 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4677712023258209 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19271919131278992 particle/accuracy_1_of_1‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_4_of_4‚0.9686647057533264 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23138584196567535 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.046995602548122406 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4480851888656616 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8645375370979309 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6400855183601379 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 epoch‚5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9920775890350342 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20873716473579407 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8685563206672668 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7691887617111206 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9944215416908264 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2194826453924179 particle/accuracy_2_of_3‚1 )loss/val_total_loss‚9.236175008933017 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28031906485557556 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9840469360351562 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9219202399253845 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7222222089767456 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7800129652023315 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7572121024131775 +jet/accuracy_3_of_3‚0.06949874013662338 (particle/accuracy‚0.8501054048538208 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28539976477622986 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0653291866183281 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 trainer/global_step‚2399 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9846875071525574 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8084046244621277 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8236206769943237 particle/accuracy_2_of_4‚1Å:¨c‘¡ ‚(:† ŠÕìÒßîߺ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eôk6Å¢ ‚(:º ŠÕìÒǘ¼ disk.sda1.in‚0 disk.dm-1.out‚ 53.33203125 disk.sda.in‚0 disk.rbd1.out‚ 4.89453125 disk.sda.out‚ 53.33203125 cpu‚26.12076648995608 disk.sda2.in‚0# disk./.usageGB‚588.734302520752 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚8037.88720703125 network.sent‚ 3.734062e+06 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 4183.78515625! disk.nvme0n1.in‚ 4183.78515625 proc.cpu.threads‚17 network.recv‚583905 disk.drbd2165.out‚0% disk.nvme0n1.out‚8037.88720703125" proc.memory.rssMB‚ 2806.4296875* proc.memory.percent‚17.129087448120117 disk.dm-0.in‚ 4183.78515625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 651.671875$ memory_percent‚96.02251052856445" disk.dm-0.out‚8037.88720703125* disk./.usagePercent‚39.510453023149964 disk.sda3.out‚ 53.33203125 disk.drbd2165.in‚0Ú%è ž£ ‚(:“ ™ÕìÒ¸ ê» gpu.0.gpu‚95.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßQCŤ ‚(:º ™ÕìÒí˜Å¼ disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚95.47953605651855 disk.rbd1.out‚ 4.89453125 disk.sda.out‚ 53.77734375' proc.memory.availableMB‚ 740.6328125 disk.dm-1.out‚ 53.77734375! disk.nvme0n1.in‚ 4183.78515625 cpu‚26.12096492102894 disk.dm-0.in‚ 4183.78515625# disk.nvme0c0n1.in‚ 4183.78515625 proc.cpu.threads‚17 network.recv‚588364 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚8352.29150390625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚8352.29150390625 network.sent‚ 3.791417e+06$ disk./.usageGB‚588.7335433959961! proc.memory.rssMB‚ 2716.765625) proc.memory.percent‚16.58182144165039 disk.sda3.out‚ 53.77734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚8352.29150390625 disk.sda.in‚0 disk.sda3.in‚0* disk./.usagePercent‚39.510402077650646 disk.sda2.in‚0 disk.dm-1.in‚0?:öО¥ ‚(:“ ¨ÕìÒéóÞ¼ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22696862 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843596033812401e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372353763483648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5272154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370438392147968 trainer/global_step‚2449 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19426256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372353746444288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82119891257344QZñ§T¬ ÙO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ “ټ΋'553967983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372353746444288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22696862 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843596033812401e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372353763483648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5272154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370438392147968 trainer/global_step‚2449 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19426256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372353746444288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82119891257344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374944816253952 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22686885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2587457400368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84739325338624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370438409187328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22696862 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2591411148800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2591069809664 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373173030724040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372501004513280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9572133 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19433888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22705928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5274137 _runtime‚2096.070563888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372501004513280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373173388313402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82153338628096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375093428296192 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433890 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695937 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84773771777536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433890 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695937 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84773771777536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373173030724040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372501004513280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9572133 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19433888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22705928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5274137 _runtime‚2096.070563888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372501004513280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373173388313402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82153338628096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0È´FI2° - _wandbruntime‚2096  _runtime‚2096ęұ ‚ /loss/FRt1/detection_loss‚0.5988331327256559 % _timestamp‚1.7843596039970653e+09 0loss/FRt1/assignment_loss‚1.9047372334265296 1loss/SRqqt1/detection_loss‚1.0044714287374568 %loss/total_loss‚9.220145089108764 trainer/global_step‚2449 _step‚201 2loss/SRqqt2/assignment_loss‚0.5687249417182368 epoch‚6 0loss/FRt2/assignment_loss‚0.8087412261789705 2loss/SRqqt1/assignment_loss‚0.6637178391113855 2loss/FBt2/assignment_loss‚0.032926884354005564 /loss/FRt2/detection_loss‚1.5526559832404405 1loss/SRqqt2/detection_loss‚0.6328029192609869 .loss/FBt1/detection_loss‚0.920340562487642 /loss/FBt2/detection_loss‚0.4983285768856776 _runtime‚2096.072226737 2loss/FBt1/assignment_loss‚0.033864360981776066¬7± ² ‡ 0loss/FRt2/assignment_loss‚0.8087412261789705 2loss/SRqqt1/assignment_loss‚0.6637178391113855 2loss/FBt2/assignment_loss‚0.032926884354005564 /loss/FRt2/detection_loss‚1.5526559832404405 1loss/SRqqt2/detection_loss‚0.6328029192609869 .loss/FBt1/detection_loss‚0.920340562487642 /loss/FBt2/detection_loss‚0.4983285768856776 _runtime‚2096.072226737 2loss/FBt1/assignment_loss‚0.033864360981776066 /loss/FRt1/detection_loss‚0.5988331327256559 % _timestamp‚1.7843596039970653e+09 0loss/FRt1/assignment_loss‚1.9047372334265296 1loss/SRqqt1/detection_loss‚1.0044714287374568 %loss/total_loss‚9.220145089108764 trainer/global_step‚2449 _step‚201 2loss/SRqqt2/assignment_loss‚0.5687249417182368 epoch‚6ÉO%D“³ ‚(:ˆ ·ÕìÒ——Ô¼ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚237.833' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0· Õ]´ ‚(:± ·ÕìÒ½š÷½' disk.nvme0c0n1.out‚8547.37255859375% disk.nvme0n1.out‚8547.37255859375( proc.memory.availableMB‚ 740.38671875! proc.memory.rssMB‚ 2716.765625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚54.71875# disk.nvme0c0n1.in‚ 4256.28515625 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0# memory_percent‚95.4810380935669) proc.memory.percent‚16.58182144165039 network.recv‚595104 disk.rbd1.in‚0 cpu‚26.124165026489095 network.sent‚ 3.801187e+06 disk.sda.out‚54.71875 disk.sda3.out‚54.71875) disk./.usa`ù§“_gePercent‚39.51039439742462! disk.nvme0n1.in‚ 4256.28515625$ disk./.usageGB‚588.7334289550781 disk.dm-0.in‚ 4256.28515625" disk.dm-0.out‚8547.37255859375 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 5.01953125 disk.rbd0.in‚04U¦Mžµ ‚(:“ ÆÕìÒÖóÛº gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉBçÀ¶ ‚(:µ ÆÕìÒ¹Ô„½# disk.nvme0c0n1.in‚ 4256.28515625$ disk.nvme0n1.out‚8555.1103515625 disk.sda2.out‚0! disk.dm-0.out‚8555.1103515625$ disk./.usageGB‚588.7334060668945 disk.rbd0.in‚0& disk.nvme0c0n1.out‚8555.1103515625 disk.sda.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 740.59765625 disk.rbd0.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 4256.28515625 disk.sda2.in‚0$ memory_percent‚95.47975063323975! proc.memory.rssMB‚ 2716.765625) proc.memory.percent‚16.58182144165039 network.sent‚ 3.866084e+06* disk./.usagePercent‚39.510392861379415 disk.dm-0.in‚ 4256.28515625 cpu‚26.12600377076383 network.recv‚599559 disk.rbd1.in‚0 disk.rbd1.out‚ 5.08203125 disk.sda3.out‚ 55.2734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 55.2734375 disk.sda.out‚ 55.2734375 disk.sda3.in‚0 proc.cpu.threads‚17f7ÿþ· ‚(:’ ÕÕìÒ“åó» gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚262.071' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—Jïd¸ ‚(:· ÕÕìÒÓý disk.drbd2165.out‚0 disk.dm-1.in‚0 network.sent‚ 3.869107e+06 disk.drbd2165.in‚0$ memory_percent‚95.48053741455078% disk.nvme0n1.out‚8560.28955078125 disk.sda3.in‚0 disk.sda2.in‚0" disk.dm-0.out‚8560.28955078125 disk.dm-1.out‚ 55.52734375 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 55.52734375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4256.28515625 disk.sda1.out‚0) proc.memory.percent‚16.58182144165039 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 4256.28515625) disk./.usagePercent‚39.51039490943969 disk.rbd1.out‚ 5.08203125% proc.memory.availableMB‚ 740.46875 cpu‚26.12766704987198$ disk./.usageGB‚588.7334365844727 disk.rbd0.in‚0! disk.nvme0n1.in‚ 4256.28515625 disk.sda3.out‚ 55.52734375! proc.memory.rssMB‚ 2716.765625 network.recv‚602878 disk.rbd1.in‚0' disk.nvme0c0n1.out‚8560.28955078125¿öe42¹ - _wandbruntime‚2135  _runtime‚2135˜âW¥Èº  _step‚202 ' lr-AdamW/pg1‚0.00042280591822877996 ' lr-AdamW/pg2‚0.00042280591822877996 trainer/global_step‚2499 % _timestamp‚1.7843596438097749e+09 _runtime‚ 2135.88514498NÝ’_Í» Ç ' lr-AdamW/pg1‚0.00042280591822877996 ' lr-AdamW/pg2‚0.00042280591822877996 trainer/global_step‚2499 % _timestamp‚1.7843596438097749e+09 _runtime‚ 2135.88514498 _step‚202Ê#oÎë2¼ - _wandbruntime‚2135  _runtime‚2135¡ÍÁÙ'½ ÓO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5375965 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2641765311488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382357921615872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19807856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638073887276 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19807858 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382357921615872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843596438108833e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379715797925888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19807858 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380402016668204 _step‚203 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379715797925888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9759463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377763942780928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2642106650624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚377763925741568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86462131866112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83791329183744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚2135.886778717 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2670784070144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83791347795968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0is„ÔÞ'¾ ØO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638073887276 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19807858 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382357921615872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843596438108833e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379715797925888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19807858 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380402016668204 _step‚203 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379715797925888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9759463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377763942780928 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2642106650624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚377763925741568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86462131866112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83791329183744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚2135.886778717 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2670784070144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83791347795968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5375965 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2641765311488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382357921615872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19807856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ë'÷—2¿ - _wandbruntime‚2136  _runtime‚21365ad§!À ÞM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19815490 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843596444236197e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23150389 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379863055994880 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4385243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382506175279616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2671809058816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377910436413440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2639084941110 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382505816893952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377910453452800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379863055994880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382506175279616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5378328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380549180804548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2671722586112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3×f¦ß=58385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2643119284736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19815488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23160380 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19815488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _step‚204 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2642777938432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520íV4é&Á ãM F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5378072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚2136.499265922 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2642777938432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19815490 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4385246 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86496480528384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382506175279616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5378328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380549180804548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2671722586112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2643119284736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19815488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23160380 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19815488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _step‚204 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2642777938432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19815490 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843596444236197e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23150389 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379863055994880 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4385243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382506175279616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2671809058816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377910436413440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2639084941110 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382505816893952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377910453452800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379863055994880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389770668599296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2692802152448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480229 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2722132530688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387778166748712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2176.162404278 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385089476374528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20189458 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9951238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692460813312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2722216913408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4470751 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616ö†6"QÐ ÙO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23605766 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385089476374528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20189458 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9951238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692460813312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2722216913408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4470751 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85462785722368 KBDæ”#PŽeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20189456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20189458 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387077866446848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23595789 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23595789 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387777809159462 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88184899640832 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389770668599296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2692802152448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480229 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2722132530688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387778166748712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2176.162404278 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387225090437120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23604841 trainer/global_step‚2549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2693814786560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387225090437120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389918922263040 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2723155798016 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20197090 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389918563877376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2723240173568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389918922263040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387225107476480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20197090 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23614832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85496195868672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385235970007040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269áZ1é&Ó ãM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23614832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85496195868672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385235970007040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20197088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88219351666688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2689701428018 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4472496 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88219454654464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387225090437120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23604841 trainer/global_step‚2549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2693814786560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387225090437120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389918922263040 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2723155798016 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20197090 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389918563877376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2723240173568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389918922263040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387225107476480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20197090Ð~~ï2Ô - _wandbruntime‚2176  _runtime‚2176¼Km~Õ þ 0loss/FRt2/assignment_loss‚0.8178216119125051 .loss/FRt1/detection_loss‚0.545362740058067 0loss/FBt2/detection_loss‚0.40573384032662335 _step‚209 2loss/SRqqt2/assignment_loss‚0.5828355115235413 %loss/total_loss‚9.532303812887022 epoch‚6 0loss/FRt1/assignment_loss‚2.0419475669671767 1loss/FBt1/assignment_loss‚0.08838733224807324 /loss/FBt2/assignment_loss‚0.031101999939277 /loss/FRt2/detection_loss‚1.5914254559416467 /loss/FBt1/detection_loss‚0.9924729837636862 % _timestamp‚1.7843596847022896e+09 2loss/SRqqt1/assignment_loss‚0.7027907723424746 0loss/SRqqt1/detection_loss‚1.099151251099582 1loss/SRqqt2/detection_loss‚0.6332727467643688 trainer/global_step‚2549 _runtime‚2176.777352274/4 Ö ƒ 1loss/SRqqt2/detection_loss‚0.6332727467643688 trainer/global_step‚2549 _runtime‚2176.777352274 0loss/FRt2/assignment_loss‚0.8178216119125051 .loss/FRt1/detection_loss‚0.545362740058067 0loss/FBt2/detection_loss‚0.40573384032662335 _step‚209 2loss/SRqqt2/assignment_loss‚0.5828355115235413 %loss/total_loss‚9.532303812887022 epoch‚6 0loss/FRt1/assignment_loss‚2.0419475669671767 1loss/FBt1/assignment_loss‚0.08838733224807324 /loss/FBt2/assignment_loss‚0.031101999939277 /loss/FRt2/detection_loss‚1.5914254559416467 /loss/FBt1/detection_loss‚0.9924729837636862 % _timestamp‚1.7843596847022896e+09 2loss/SRqqt1/assignment_loss‚0.7027907723424746 0loss/SRqqt1/detection_loss‚1.099151251099582Ñœ¾oŽ’× ‚(:‡ ‘ÖìÒòëÏ» gpu.0.gpu‚45.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rkGÀØ ‚(:µ ‘ÖìÒý»Ù½ disk.dm-0.in‚ 4315.6953125 disk.rbd1.out‚ 5.20703125' disk.nvme0c0n1.out‚8612.84619140625 disk.sda3.in‚0 disk.drbd2165.out‚0 network.sent‚ 3.94675e+06 disk.nvme0n1.in‚ 4315.6953125 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 57.18359375 cpu‚26.13368670993437# memory_percent‚95.4822301864624) disk./.usagePercent‚39.56038601266586 disk.rbd1.in‚0% disk.nvme0n1.out‚8612.84619140625 disk.sda1.in‚0" disk.dm-0.out‚8612.84619140625 disk.sda.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 4315.6953125! proc.memory.rssMB‚ 2716.765625) proc.memory.percent‚16.58182144165039 proc.cpu.threads‚17( proc.memory.availableMB‚ 740.19140625 network.recv‚617857$ disk./.usageGB‚589.4783401489258 disk.rbd0.out‚0 disk.sda.out‚ 57.18359375 disk.sda3.out‚ 57.18359375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0úŒ(n“Ù ‚(:ˆ  ÖìÒ󧔺 gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t<®VÀÚ ‚(:µ  ÖìÒÍæ½$ disk./.usageGB‚589.4756622314453 disk.sda3.out‚ 57.4453125 network.recv‚622465 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚9405.00439453125( proc.memory.percent‚16.5818452835083' disk.nvme0c0n1.out‚9405.00439453125 disk.sda2.in‚0 disk.dm-0.in‚ 4315.6953125 disk.dm-1.out‚ 57.4453125 network.sent‚ 4.009671e+06$ memory_percent‚95.48227787017822) disk./.usagePercent‚39.56020629537682 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.26953125 disk.nvme0n1.in‚ 4315.6953125 disk.sda.out‚ 57.4453125 disk.sda2.out‚0# proc.memory.rssMB‚ 2716.76953125 cpu‚26.13481061629088% disk.nvme0n1.out‚9405.00439453125( proc.memory.availableMB‚ 740.18359375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4315.6953125 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17ªÏ2Û - _wandbruntime‚2216  _runtime‚2216cnÆÜ À & lr-AdamW/pg2‚0.0004216287022325533 trainer/global_step‚2599 $ _timestamp‚1.784359724537773e+09 _runtime‚2216.613085259 _step‚210 & lr-AdamW/pg1‚0.0004216287022325533{‰PËÝ Å & lr-AdamW/pg2‚0.0004216287022325533 trainer/global_step‚2599 $ _timestamp‚1.784359724537773e+09 _runtime‚2216.613085259 _step‚210 & lr-AdamW/pg1‚0.0004216287022325533Ò^ž2Þ - _wandbruntime‚2216  _runtime‚2216¤ÃcO* ß ÖO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2743497654272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20571056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24060218 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4558001 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89907573500928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7843597245388618e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDevice ÿѲStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2743156315136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2773452847104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395154316829220 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2743156315136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397183415582720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20571056 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24060218 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395153959239970 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392415009968128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394439917928448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24050241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394439917928448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2773368464384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚211 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚2599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2738966311202 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397183057204224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10142013 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394439900889088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24050241 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5584270 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20571058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89907676495872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394439917928448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2773368464384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚211 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚2599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2738966311202 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397183057204224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10142013 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394439900889088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24050241 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5584270 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20571058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89907676495872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2743156315136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2773452847104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395154316829220 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2743156315136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250Ó­[]Ë2á - _wandbruntime‚2217  _runtime‚2217ÈóYää&â ÞM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚212 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24059293 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397331669246464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394587158958080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394587141918720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89942131938816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87167652407296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392561503600640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚2217.226117134 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20578690 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774479531520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24059293 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2744510288384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89942026853888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395301838554926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20578690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4559746 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394587141918720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397331669246464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4559743 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2744168942080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0.Bd4ã ãM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2744168942080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚212 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24059293 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397331669246464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394587158958080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394587141918720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89942131938816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87167652407296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392561503600640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚2217.226117134 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 UEDeviceSNË.aµtatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20578690 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774479531520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24059293 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2744510288384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89942026853888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395301838554926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20578690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4559746 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394587141918720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397331669246464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4559743 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983Ô1Âö2ä - _wandbruntime‚2217  _runtime‚22178å þ /loss/FBt2/detection_loss‚0.7055948218408351 trainer/global_step‚2599 2loss/SRqqt2/assignment_loss‚0.5269671323844737 % _timestamp‚1.7843597251534498e+09 _runtime‚2217.228509861 _step‚213 0loss/FRt2/assignment_loss‚0.7279629034760557 0loss/FBt1/assignment_loss‚0.1742235984331108 %loss/total_loss‚9.864426635437958 /loss/FRt1/assignment_loss‚1.972428723179936 2loss/FBt2/assignment_loss‚0.028512607079499448 /loss/FRt2/detection_loss‚1.4840584920075388 /loss/FBt1/detection_loss‚1.3616462558169256 epoch‚6 2loss/SRqqt1/assignment_loss‚0.6263910745340556 /loss/FRt1/detection_loss‚0.5687342466142472 0loss/SRqqt1/detection_loss‚1.015912526859776 0loss/SRqqt2/detection_loss‚0.671994253211505W7K æ ƒ 2loss/SRqqt2/assignment_loss‚0.5269671323844737 % _timestamp‚1.7843597251534498e+09 _runtime‚2217.228509861 _step‚213 0loss/FRt2/assignment_loss‚0.7279629034760557 0loss/FBt1/assignment_loss‚0.1742235984331108 %loss/total_loss‚9.864426635437958 /loss/FRt1/assignment_loss‚1.972428723179936 2loss/FBt2/assignment_loss‚0.028512607079499448 /loss/FRt2/detection_loss‚1.4840584920075388 /loss/FBt1/detection_loss‚1.3616462558169256 epoch‚6 2loss/SRqqt1/assignment_loss‚0.6263910745340556 /loss/FRt1/detection_loss‚0.5687342466142472 0loss/SRqqt1/detection_loss‚1.015912526859776 0loss/SRqqt2/detection_loss‚0.671994253211505 /loss/FBt2/detection_loss‚0.7055948218408351 trainer/global_step‚2599ÕÉœj“ç ‚(:ˆ ¯ÖìÒÏÖ­» gpu.0.gpu‚41.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙdÞÃè ‚(:¸ ¯ÖìÒÿßÔ¼ disk.sda3.out‚ 57.92578125 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚26.136714851186877" disk.nvme0c0n1.in‚ 4368.4609375% disk.nvme0n1.out‚9407.46630859375 disk.dm-1.in‚0) proc.memory.percent‚16.58186912536621 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.out‚ 57.92578125" proc.memory.rssMB‚ 2716.7734375 disk.sda.out‚ 57.92578125 network.recv‚626191) disk./.usagePercent‚39.51049321633284 disk.rbd1.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 740.1953125 disk.rbd0.out‚0$ memory_percent‚95.48220634460449 disk.dm-0.in‚ 4368.4609375" disk.dm-0.out‚9407.46630859375 network.sent‚ 4.014592e+06$ disk./.usageGB‚588.7349014282227 disk.rbd1.out‚ 5.26953125 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚9407.46630859375 disk.nvme0n1.in‚ 4368.4609375 disk.sda3.in‚0Ùx‘‹žé ‚(:“ ¾ÖìÒ‹™†» gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\çžT»ê ‚(:° ¾ÖìÒꟽ disk.rbd1.out‚ 5.39453125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4368.4609375$ disk.nvme0c0n1.out‚ 9413.08984375 disk.sda1.out‚0) proc.memory.percent‚16.58186912536621 cpu‚26.137790471470623 network.recv‚630364 disk.nvme0n1.in‚ 4368.4609375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.out‚ 9413.08984375 disk.dm-1.out‚ 58.34765625 disk.sda.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚ 4368.4609375( proc.memory.availableMB‚ 740.26171875) disk./.usagePercent‚39.50870807579658$ disk./.usageGB‚588.7083015441895 disk.rbd1.in‚0" disk.nvme0n1.out‚ 9413.08984375 disk.sda.out‚ 58.34765625 network.sent‚ 4.077587e+06 disk.rbd0.in‚0 disk.sda3.out‚ 58.34765625 disk.sda2.in‚0" proc.memory.rssMB‚ 2716.7734375 disk.dm-1.in‚0 disk.drbd2165.out‚0$ memory_percent‚95.48180103302002¿‚™´ë ‚(:© ÍÖìÒøœ´¼ disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 4368.4609375 disk.sda.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4368.4609375$ memory_percent‚96.98572158813477 network.recv‚633784 disk.drbd2165.out‚0 disk.dm-1.out‚60.46875& disk.nvme0c0n1.out‚9420.9951171875 disk.sda1.out‚0" proc.memory.rssMB‚ 2716.7734375! disk.dm-0.out‚9420.9951171875 disk.dm-1.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.58186912536621 disk.sda.out‚60.46875' disk./.usagePercent‚39.508673770787 disk.rbd1.in‚0 disk.rbd1.out‚ 5.39453125 disk.sda3.in‚0 disk.sda3.out‚60.46875$ disk./.usageGB‚588.7077903747559 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4368.4609375$ disk.nvme0n1.out‚9420.9951171875 cpu‚26.139644260475727& proc.memory.availableMB‚ 493.859375 network.sent‚ 4.082452e+06 disk.sda1.in‚0kJæÚì ‚(:… ÍÖìÒÞ¼ê½ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒAÜL2í - _wandbruntime‚2257  _runtime‚2257?“Ó9Çî Á & lr-AdamW/pg2‚0.0004210204089022727 trainer/global_step‚2649 % _timestamp‚1.7843597650577986e+09 _runtime‚2257.133112199 _step‚214 & lr-AdamW/pg1‚0.0004210204089022727^ øÌï Æ _runtime‚2257.133112199 _step‚214 & lr-AdamW/pg1‚0.0004210204089022727 & lr-AdamW/pg2‚0.0004210204089022727 trainer/global_step‚2649 % _timestamp‚1.7843597650577986e+09Ö|i/ª2ð - _wandbruntime‚2257  _runtime‚2257Þ·¶ËÛ'ñ ÕO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚401801952370688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2789923348000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24504693 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88805661575168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2794193156096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91630385042944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚404596162566144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2257.134466687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404596162566144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚404595804187648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2824742080000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402530109320478 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚2649 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399740543561728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24514670 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91630488037888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2824826462720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24514670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4645248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20952658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5687994 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10332964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚401801952370688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401801969410048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10333245 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784359765058914e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚401801969410048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20952656 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2793851816960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20952656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402530466909728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4645251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5687716 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2793851816960 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2789923348000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24504693 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88805661575168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2794193156096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91630385042944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚404596162566144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2257.134466687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404596162566144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚404595804187648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2824742080000 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402530109320478 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚2649 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399740543561728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24514670 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91630488037888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2824826462720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24514670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4645248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20952658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5687994 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10332964 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚401801952370688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401801969410048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10333245 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784359765058914e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚401801969410048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20952656 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2793851816960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20952656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402530466909728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4645251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5687716׶à2ó - _wandbruntime‚2257  _runtime‚2257àÜÀ·ô ÞM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚2649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2257.747679538 XJDeviceStatsMonitor.on_train_ba~§¼-#tch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399887037194240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2794864443904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91664831324160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404744416229888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401949210439680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404744057844224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2790593851832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2795205790208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24523736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20960290 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88839071721472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591Î¥“¨é&õ ãM _runtime‚2257.747679538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399887037194240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2794864443904 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91664831324160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404744416229888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401949210439680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404744057844224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2790593851832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2795205790208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24523736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20960290 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88839071721472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚2649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ø”‡?2ö - _wandbruntime‚2257  _runtime‚2257X&m÷ ü epoch‚6 /loss/FRt1/assignment_loss‚1.945988993404964 2loss/SRqqt2/assignment_loss‚0.6508508367779544 1loss/FBt1/assignment_loss‚0.10142512627256413 1loss/FBt2/assignment_loss‚0.08263659511479052 _runtime‚ 2257.74960854 1loss/SRqqt2/detection_loss‚0.6417014979814742 trainer/global_step‚2649 /loss/FBt2/detection_loss‚0.6245694359009765 %loss/total_loss‚9.628594972588278 2loss/SRqqt1/assignment_loss‚0.6684577496738972 /loss/FRt1/detection_loss‚0.5114798782263886 .loss/FRt2/detection_loss‚1.629825460748051 .loss/FBt1/detection_loss‚0.802635162792824 % _timestamp‚1.7843597656743608e+09 /loss/FRt2/assignment_loss‚0.855059264133675 1loss/SRqqt1/detection_loss‚1.1139649715607187 _step‚217f’VAø  1loss/FBt2/assignment_loss‚0.08263659511479052 _runtime‚ 2257.74960854 1loss/SRqqt2/detection_loss‚0.6417014979814742 trainer/global_step‚2649 /loss/FBt2/detection_loss‚0.6245694359009765 %loss/total_loss‚9.628594972588278 2loss/SRqqt1/assignment_loss‚0.6684577496738972 /loss/FRt1/detection_loss‚0.5114798782263886 .loss/FRt2/detection_loss‚1.629825460748051 .loss/FBt1/detection_loss‚0.802635162792824 % _timestamp‚1.7843597656743608e+09 /loss/FRt2/assignment_loss‚0.855059264133675 1loss/SRqqt1/detection_loss‚1.1139649715607187 _step‚217 epoch‚6 /loss/FRt1/assignment_loss‚1.945988993404964 2loss/SRqqt2/assignment_loss‚0.6508508367779544 1loss/FBt1/assignment_loss‚0.10142512627256413ÙL½“ù ‚(:ˆ ÜÖìÒùÛº gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îÇf’¿ú ‚(:´ ÜÖìÒ©þì¼' disk./.usagePercent‚39.508673770787 disk.rbd1.out‚ 5.45703125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 60.97265625 disk.sda2.out‚0% proc.memory.availableMB‚ 493.78125$ disk./.usageGB‚588.7077903747559% disk.nvme0n1.out‚9429.49169921875$ memory_percent‚96.98619842529297 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 60.97265625 network.sent‚ 4.087384e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚9429.49169921875" proc.memory.rssMB‚ 2716.7734375 cpu‚26.141230374088277 disk.dm-1.in‚0 network.recv‚636687 disk.sda.out‚ 60.97265625 disk.sda2.in‚0 disk.dm-0.in‚ 4368.4609375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4368.4609375 disk.nvme0n1.in‚ 4368.4609375) proc.memory.percent‚16.58186912536621 proc.cpu.threads‚17 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9429.49169921875 disk.sda1.in‚0—=Ue’û ‚(:‡ ëÖìÒþ¸Æ» gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¼®Å»ü ‚(:° ëÖìÒÇÿ¶¼ disk.sda2.in‚0" disk.dm-0.out‚9432.37255859375 disk.dm-1.out‚ 61.328125" disk.nvme0c0n1.in‚ 4414.1015625' disk.nvme0c0n1.out‚9432.37255859375 cpu‚26.143141764357807 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 4414.1015625 disk.rbd1.out‚ 5.45703125" proc.memory.rssMB‚ 2716.7734375) proc.memory.percent‚16.58186912536621 disk.sda2.out‚0 disk.drbd2165.out‚0$ memory_percent‚96.98657989501953 disk.sda3.out‚ 61.328125% proc.memory.availableMB‚ 493.71875) disk./.usagePercent‚39.50855523929864 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 network.recv‚641077$ disk./.usageGB‚588.7060241699219% disk.nvme0n1.out‚9432.37255859375 disk.sda.out‚ 61.328125 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 4.150146e+06 disk.nvme0n1.in‚ 4414.1015625ÐFÏ@“ý ‚(:ˆ úÖìÒÇ×» gpu.0.gpu‚85.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸÷wZ·þ ‚(:¬ úÖìÒ¸šì¼ disk.nvme0n1.in‚ 4414.1015625 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚61.9375$ memory_percent‚96.98653221130371 network.recv‚644387 disk.rbd0.out‚0 disk.sda.out‚61.9375 disk.sda3.out‚61.9375 proc.cpu.threads‚17) disk./.usagePercent‚39.50856522359247 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 4414.1015625" disk.dm-0.out‚9436.89794921875$ disk./.usageGB‚588.7061729431152 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 4414.1015625 cpu‚26.144361952131128' disk.nvme0c0n1.out‚9436.89794921875 disk.sda2.out‚0' proc.memory.availableMB‚ 493.7265625) proc.memory.percent‚16.58186912536621 disk.rbd1.out‚ 5.51953125% disk.nvme0n1.out‚9436.89794921875 disk.sda.in‚0" proc.memory.rssMB‚ 2716.7734375 network.sent‚ 4.155113e+06 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0X‡Š2ÿ - _wandbruntime‚2297  _runtime‚2297Óç²Æ€ À & lr-AdamW/pg1‚0.0004203990438292449 & lr-AdamW/pg2‚0.0004203990438292449 trainer/global_step‚2699 % _timestamp‚1.7843598053588486e+09 _runtime‚ 2297.43423297 _step‚218Îq$Ë Å & lr-AdamW/pg1‚0.0004203990438292449 & lr-AdamW/pg2‚0.0004203990438292449 trainer/global_step‚2699 % _timestamp‚1.7843598053588486e+09 _runtime‚ 2297.43423297 _step‚218Ú?ì˜92‚ - _wandbruntime‚2297  _runtime‚2297%ú×Øaƒ ÖO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93353149074944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21334256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409164003852288 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10524026 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843598053600295e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409164003852288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21334256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90477099501568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24969122 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409906616990236 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407066060115968 trainer/global_step‚2699 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21334258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2844888657920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412008909549568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5791254 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2876049573376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24959145 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90477080889344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚407066077155328 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24969122 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412008551171072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10523752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2844547318784 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93353043982848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚2297.435763489 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46293401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21334258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409164020891648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2844888657920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4732498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2875963093504 C:DeviceStatsMonitor.on_train_batch_start/activeKmØ{ .all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚219 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2840199285018 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2844547318784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2844547318784 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93353043982848 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚2297.435763489 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46293401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21334258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409164020891648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2844888657920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4732498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2875963093504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚219 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2840199285018 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2844547318784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2876049573376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24959145 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90477080889344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ût@ÒÏ2… - _wandbruntime‚2298  _runtime‚2298Ù'8ä&† ÞM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2841210338740 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21341888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2841550888742 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24968197 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 # _timestamp‚1.78435980597575e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2845559945728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5793570 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90510509647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412157163213312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2845559945728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4734246 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412156804827648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845901292032 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409311244881920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2877067443200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845901292032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2699 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚220 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412157163213312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407212570787840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93387595703296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21341890 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2298.051310863 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409311261921280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21341888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407212587827200 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2699 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚220 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412157163213312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407212570787840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93387595703296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21341890 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2298.051310863 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409311261921280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21341888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407212587827200 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2845559945728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5793570 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90510509647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412157163213312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2845559945728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4734246 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412156804827648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845901292032 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Üö´ž±2ˆ - _wandbruntime‚2298  _runtime‚2298;½¦‰  1loss/SRqqt1/detection_loss‚1.0479337161505886 % _timestamp‚1.7843598059778256e+09 _runtime‚2298.052904161 2loss/SRqqt1/assignment_loss‚0.6320276323920951 1loss/FBt1/assignment_loss‚0.08878988672322426 .loss/FRt2/detection_loss‚1.575782290644858 %loss/total_loss‚9.413822471201108 trainer/global_step‚2699 _step‚221 0loss/FRt1/assignment_loss‚1.8566366746841139 2loss/SRqqt2/assignment_loss‚0.6023335386393353 /loss/FRt1/detection_loss‚0.5630902672478217 1loss/SRqqt2/detection_loss‚0.7000071151948797 epoch‚6 0loss/FRt2/assignment_loss‚0.8166201411992654 2loss/FBt2/assignment_loss‚0.024155966499558496 /loss/FBt1/detection_loss‚1.0254554870848855 /loss/FBt2/detection_loss‚0.4809897547404814ÒƒO Š † % _timestamp‚1.7843598059778256e+09 _runtime‚2298.052904161 2loss/SRqqt1/assignment_loss‚0.6320276323920951 1loss/FBt1/assignment_loss‚0.08878988672322426 .loss/FRt2/detection_loss‚1.575782290644858 %loss/total_loss‚9.413822471201108 trainer/global_step‚2699 _step‚221 0loss/FRt1/assignment_loss‚1.8566366746841139 2loss/SRqqt2/assignment_loss‚0.6023335386393353 /loss/FRt1/detection_loss‚0.5630902672478217 1loss/SRqqt2/detection_loss‚0.7000071151948797 epoch‚6 0loss/FRt2/assignment_loss‚0.8166201411992654 2loss/FBt2/assignment_loss‚0.024155966499558496 /loss/FBt1/detection_loss‚1.0254554870848855 /loss/FBt2/detection_loss‚0.4809897547404814 1loss/SRqqt1/detection_loss‚1.0479337161505886ÝP\SR‹ ‚(:’ ‰×ìÒ‰¹ñº gpu.0.gpu‚30.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 #ˆÃŒ ‚(:¸ ‰×ìÒž„ß¼) disk./.usagePercent‚39.50862564137056 disk.rbd1.out‚ 5.51953125' disk.nvme0c0n1.out‚9439.51611328125 disk.sda3.out‚ 62.20703125 disk.drbd2165.in‚0" disk.dm-0.out‚9439.51611328125 disk.dm-1.in‚0$ memory_percent‚96.98789119720459$ disk./.usageGB‚588.7070732116699) proc.memory.percent‚16.58186912536621 proc.cpu.threads‚17 disk.rbd1.in‚0% disk.nvme0n1.out‚9439.51611328125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚647830 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 4414.1015625 disk.nvme0n1.in‚ 4414.1015625 disk.dm-0.in‚ 4414.1015625" proc.memory.rssMB‚ 2716.7734375 cpu‚26.14482904353833 network.sent‚ 4.160231e+06 disk.dm-1.out‚ 62.20703125 disk.rbd0.in‚0 disk.sda.out‚ 62.20703125 disk.drbd2165.out‚0( proc.memory.availableMB‚ 493.50390625 disk.sda1.out‚0Ö$ò“ž ‚(:“ ˜×ìÒÒÀî» gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.408' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.281600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó+©ÐÄŽ ‚(:¹ ˜×ìÒðï˼ disk.rbd0.out‚0 disk.sda.in‚0) proc.memory.percent‚16.58186912536621$ disk./.usageGB‚588.7071762084961 disk.sda2.out‚0 disk.sda.out‚ 62.65234375 network.recv‚652084 disk.dm-1.in‚0 disk.rbd1.out‚ 5.58203125 disk.rbd0.in‚0 disk.sda1.in‚0) disk./.usagePercent‚39.50863255357398 disk.dm-1.out‚ 62.65234375 disk.nvme0n1.in‚ 4414.1015625$ memory_percent‚96.99175357818604 disk.sda2.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚9444.32080078125% disk.nvme0n1.out‚9444.32080078125 disk.sda3.in‚0 disk.sda1.out‚0 network.sent‚ 4.223017e+06 disk.drbd2165.out‚0 disk.dm-0.in‚ 4414.1015625' disk.nvme0c0n1.out‚9444.32080078125" proc.memory.rssMB‚ 2716.7734375 cpu‚26.144014407527898 proc.cpu.threads‚17 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 4414.1015625 disk.sda3.out‚ 62.65234375( proc.memory.availableMB‚ 492.87109375{›Ç†2 - _wandbruntime‚2338  _runtime‚23382×É Ã _runtime‚2338.367116021 _step‚222 ' lr-AdamW/pg1‚0.00041976464692258726 ' lr-AdamW/pg2‚0.00041976464692258726 trainer/global_step‚2749 % _timestamp‚1.7843598462917964e+09XökøΑ È ' lr-AdamW/pg1‚0.00041976464692258726 ' lr-AdamW/pg2‚0.00041976464692258726 trainer/global_step‚2749 % _timestamp‚1.7843598462917964e+09 _runtime‚2338.367116021 _step‚222Þ„³4`2’ - _wandbruntime‚2338  _runtime‚2338;±üÜ'“ ÖO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927004111872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843598462926278e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21715856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416526072373248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10713570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419421656532992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2895242820608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21715858 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2891156321816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47129453 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2895242820608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21715856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5894079 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95075636408320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92148518815744 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21715858 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4819748 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25423574 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚416526055333888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚2338.368246135 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417282767070744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2895584159744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25413597 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10713830 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95075522927616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414391610748928 trainer/global_step‚2749 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21715858 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4819748 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25423574 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚416526055333888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚2338.368246135 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417282767070744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2895584159744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25413597 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10713830 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95075522927616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414391610748928 trainer/global_step‚2749 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92181947574272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414538121420800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5895879 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92181966186496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2928032437760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21723488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414538104381440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25422649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2749 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21723490 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2892167375650 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25432640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843598469079804e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419569910196736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚416673313402880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚2338.983600947 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2928121007616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10717631 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419569910196736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0µÓÔœë&— åM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414538121420800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5895879 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92181966186496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2928032437760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21723488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414538104381440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25422649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2749 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21723490 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2892167375650 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25432640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843598469079804e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419569910196736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚416673313402880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚2338.983600947 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2928121007616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10717631 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419569910196736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2896255447552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92181947574272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856àý†2˜ - _wandbruntime‚2338  _runtime‚2338Ówž6™  0loss/FRt1/assignment_loss‚2.0340395642393094 2loss/SRqqt1/assignment_loss‚0.6721766826477693 /loss/FBt1/detection_loss‚1.5042357084651787 &loss/total_loss‚10.358590453453056 trainer/global_step‚2749 _runtime‚2338.985974358 /loss/FRt1/detection_loss‚0.5585811812859064 0loss/FRt2/assignment_loss‚0.7872278563360822 2loss/SRqqt2/assignment_loss‚0.5980375365038136 0loss/FBt1/assignment_loss‚0.2612595519999622 1loss/FBt2/assignment_loss‚0.07423309162511046 /loss/FRt2/detection_loss‚1.5610397854273705 1loss/SRqqt1/detection_loss‚1.0853479285096974 1loss/SRqqt2/detection_loss‚0.6475574703944125 /loss/FBt2/detection_loss‚0.5748540960184441 epoch‚6 % _timestamp‚1.7843598469108655e+09 _step‚2258a¦ô š † 0loss/FRt1/assignment_loss‚2.0340395642393094 2loss/SRqqt1/assignment_loss‚0.6721766826477693 /loss/FBt1/detection_loss‚1.5042357084651787 &loss/total_loss‚10.358590453453056 trainer/global_step‚2749 _runtime‚2338.985974358 /loss/FRt1/detection_loss‚0.5585811812859064 0loss/FRt2/assignment_loss‚0.7872278563360822 2loss/SRqqt2/assignment_loss‚0.5980375365038136 0loss/FBt1/assignment_loss‚0.2612595519999622 1loss/FBt2/assignment_loss‚0.07423309162511046 /loss/FRt2/detection_loss‚1.5610397854273705 1loss/SRqqt1/detection_loss‚1.0853479285096974 1loss/SRqqt2/detection_loss‚0.6475574703944125 /loss/FBt2/detection_loss‚0.5748540960184441 epoch‚6 % _timestamp‚1.7843598469108655e+09 _step‚225áƒÚz™¹› ‚(:® §×ìÒ̧·¼ disk.dm-1.out‚ 63.3671875 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 492.5703125 network.recv‚655619$ disk./.usageGB‚588.7084922790527 disk.rbd0.in‚0 network.sent‚ 4.227998e+06) disk./.usagePercent‚39.50872087617329% disk.nvme0n1.out‚9448.13623046875 disk.sda3.out‚ 63.3671875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 4478.21875) proc.memory.percent‚16.58186912536621" disk.dm-0.out‚9448.13623046875 disk.sda1.in‚0$ memory_percent‚96.99358940124512 disk.sda.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4478.21875' disk.nvme0c0n1.out‚9448.13623046875 disk.nvme0n1.in‚ 4478.21875 disk.rbd1.out‚ 5.58203125 disk.sda.out‚ 63.3671875 disk.sda3.in‚0" proc.memory.rssMB‚ 2716.7734375 cpu‚26.14321061399577 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0Ï?“œ ‚(:ˆ §×ìÒ¹ïʽ gpu.0.gpu‚41.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÎY“ ‚(:ˆ ¶×ìÒá²» gpu.0.gpu‚69.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Õ‹F·ž ‚(:¬ ¶×ìÒ㦽) disk./.usagePercent‚39.50872471628631 disk.sda1.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 492.421875) proc.memory.percent‚16.58186912536621$ disk./.usageGB‚588.7085494995117 disk.dm-0.in‚ 4478.21875 disk.dm-1.out‚ 63.74609375 disk.rbd0.in‚0 disk.sda.out‚ 63.74609375 network.recv‚660144 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2716.7734375 cpu‚26.14228767907445 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0# memory_percent‚96.9944953918457! disk.dm-0.out‚9454.7451171875 network.sent‚ 4.290915e+06 disk.rbd1.out‚ 5.64453125& disk.nvme0c0n1.out‚9454.7451171875 disk.nvme0n1.in‚ 4478.21875 disk.sda3.out‚ 63.74609375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4478.21875$ disk.nvme0n1.out‚9454.7451171875æÇÕ¤Ÿ ‚(:’ Å×ìÒ¾±Ú» gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÅÃã±  ‚(:¦ Å×ìÒ§‚ν* disk./.usagePercent‚39.508722156210965 disk.dm-0.in‚ 4478.21875 disk.dm-1.out‚ 71.5703125 disk.rbd1.out‚ 5.64453125 disk.sda.out‚ 71.5703125 cpu‚26.141494049247306 disk.dm-0.out‚ 9458.99609375 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4478.21875 disk.nvme0n1.in‚ 4478.21875" disk.nvme0n1.out‚ 9458.99609375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0& proc.memory.availableMB‚ 492.859375 disk.sda3.out‚ 71.5703125$ memory_percent‚96.99182510375977 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 network.recv‚663568$ disk.nvme0c0n1.out‚ 9458.99609375" proc.memory.rssMB‚ 2716.7734375) proc.memory.percent‚16.58186912536621 network.sent‚ 4.297833e+06$ disk./.usageGB‚588.7085113525391 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.in‚0Ylé<2¡ - _wandbruntime‚2379  _runtime‚2379­˜¹…È¢  ' lr-AdamW/pg1‚0.00041911725892842784 ' lr-AdamW/pg2‚0.00041911725892842784 trainer/global_step‚2799 $ _timestamp‚1.784359886964751e+09 _runtime‚2379.040097157 _step‚226Ÿ8ÕÍ£ Ç ' lr-AdamW/pg2‚0.00041911725892842784 trainer/global_step‚2799 $ _timestamp‚1.784359886964751e+09 _runtime‚2379.040097157 _step‚226 ' lr-AdamW/pg1‚0.00041911725892842784âÅõBã2¤ - _wandbruntime‚2379  _runtime‚2379sFÓÜ'¥ ÖO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424658559562002 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96798156531200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2941772808724 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424658917151252 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93819975354368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚2379.041350002 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2946279661568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426834403516416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10904907 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423888106815488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22097456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2978199789056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5997906 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2945938322432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426834045137920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10904654 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421717144342528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426834403516416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423888123854848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421717127303168 _step‚227 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421717144342528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426834403516416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423888123854848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421717127303168 _step‚227 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424658917151252 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93819975354368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚2379.041350002 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2946279661568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426834403516416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10904907 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906998 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423888106815488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22097456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2978199789056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5997906 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2945938322432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426834045137920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10904654 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0㱂۩2§ - _wandbruntime‚2379  _runtime‚2379'¤±æ&¨ àM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2942783862558 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚424035347845120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93853404112896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2947292295680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚228 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424035364884480 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4908743 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2799 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93853385500672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426982298794496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2942443312556 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10908684 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22105088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96832716435456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5999938 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2379.653064388 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22105090 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2946950949376 % _timestamp‚1.7843598875775235e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304àr‘Çë&© åM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426982298794496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2942443312556 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10908684 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22105088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96832716435456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5999938 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2379.653064388 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22105090 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2946950949376 % _timestamp‚1.7843598875775235e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2942783862558 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚424035347845120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93853404112896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2947292295680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚228 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424035364884480 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4908743 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚2799 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93853385500672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684ä3v—×2ª - _wandbruntime‚2379  _runtime‚2379qÏ™«  0loss/FRt2/assignment_loss‚0.7529157747582691 2loss/SRqqt1/assignment_loss‚0.6777106306974637 2loss/SRqqt2/assignment_loss‚0.5832384137345231 %loss/total_loss‚9.876112625581198 epoch‚6 1loss/FBt2/assignment_loss‚0.07510184878278817 /loss/FRt1/detection_loss‚0.5408751103621896 /loss/FRt2/detection_loss‚1.6100134776132862 1loss/SRqqt2/detection_loss‚0.6840176449361718 /loss/FBt1/detection_loss‚1.2058391126875694 trainer/global_step‚2799 $ _timestamp‚1.784359887579924e+09 _runtime‚2379.655006184 _step‚229 2loss/FBt1/assignment_loss‚0.031641361483506304 1loss/SRqqt1/detection_loss‚1.1618535718256668 /loss/FBt2/detection_loss‚0.5345487315994752 0loss/FRt1/assignment_loss‚2.0183569471002896€›dD ¬ † 2loss/SRqqt2/assignment_loss‚0.5832384137345231 %loss/total_loss‚9.876112625581198 epoch‚6 1loss/FBt2/assignment_loss‚0.07510184878278817 /loss/FRt1/detection_loss‚0.5408751103621896 /loss/FRt2/detection_loss‚1.6100134776132862 1loss/SRqqt2/detection_loss‚0.6840176449361718 /loss/FBt1/detection_loss‚1.2058391126875694 trainer/global_step‚2799 $ _timestamp‚1.784359887579924e+09 _runtime‚2379.655006184 _step‚229 2loss/FBt1/assignment_loss‚0.031641361483506304 1loss/SRqqt1/detection_loss‚1.1618535718256668 /loss/FBt2/detection_loss‚0.5345487315994752 0loss/FRt1/assignment_loss‚2.0183569471002896 0loss/FRt2/assignment_loss‚0.7529157747582691 2loss/SRqqt1/assignment_loss‚0.6777106306974637åüµçB±­ ‚(:¦ Ô×ìÒ¸®¼ disk.sda3.in‚0 disk.dm-0.in‚ 4478.21875* proc.memory.percent‚16.483402252197266 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 81.4296875 disk.sda1.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4478.21875# disk.nvme0n1.out‚9465.755859375# memory_percent‚96.8958854675293 disk.rbd0.in‚0% disk.nvme0c0n1.out‚9465.755859375 disk.sda2.out‚0! proc.memory.rssMB‚ 2700.640625 disk.rbd0.out‚0 disk.sda.out‚ 81.4296875 cpu‚26.15019500466086 disk.sda3.out‚ 81.4296875 disk.sda2.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.recv‚666925) disk./.usagePercent‚39.50872906841439 disk.nvme0n1.in‚ 4478.21875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.out‚9465.755859375& proc.memory.availableMB‚ 508.578125 network.sent‚ 4.300797e+06$ disk./.usageGB‚588.7086143493652 disk.rbd1.out‚ 5.70703125&= 6® ‚(:… Ô×ìÒ­ø» gpu.0.gpu‚16.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÅW2¯ - _wandbruntime‚2398  _runtime‚2398KŽŸ^ò ° »{ 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6712158918380737 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4597400426864624 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.38250628113746643 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8078430891036987 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11333334445953369 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8388831615447998 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22398917376995087 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02735362946987152 particle/accuracy_4_of_5‚1 )loss/val_total_loss‚9.127449575498366 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04047080874443054 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5337244272232056 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7209983468055725 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_3‚0.05911188945174217 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13009274005889893 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21038818359375 % _timestamp‚1.7843599059705105e+09 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.399338036775589 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941222071647644 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.767119288444519 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4736865758895874 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044114671647548676 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2194276601076126 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03627071902155876 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 jet/accuracy_4_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18723881244659424 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22371704876422882 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19921515882015228 >&Purity/1FRt*SRqqt1FBt/evenCp©Ï1t_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21661460399627686 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6565626859664917 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3778187930583954 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692741394042969 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6917603611946106 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8818972706794739 *jet/accuracy_1_of_2‚0.7985442876815796 *jet/accuracy_2_of_3‚0.5379233360290527 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17669321596622467 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19751830399036407 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6774737238883972 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9937629699707031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9016722440719604 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8818594813346863 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952654242515564 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7144115567207336 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8826342225074768 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6760363578796387 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6746310591697693 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6526148319244385 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23053103685379028 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951386451721191 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7800557017326355 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7651753425598145 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22398917376995087 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚2799 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6309691071510315 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0993061289191246 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6834532022476196 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_2_of_2‚0.14172162115573883 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7319905757904053 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.765900731086731 (particle/accuracy‚0.8433315753936768 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06356984376907349 particle/accuracy_5_of_5‚0 _runtime‚2398.046555724 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996243953704834 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7665168046951294 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7180811166763306 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6770930886268616 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5087500214576721 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.848517119884491 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6749030351638794 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03947584331035614 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48134884238243103 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9856293797492981 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6770930886268616 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.540668785572052 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03986300155520439 *jet/accuracy_1_of_3‚0.9283366203308105 particle/accuracy_3_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5887508988380432 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11271053552627563 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1706826388835907 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7993576526641846 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19351625442504883 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7663083672523499 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859439134597778 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23724229633808136 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03627071902155876 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9298042058944702 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.760002076625824 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.697464644908905 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9845884442329407 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1994244009256363 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.777563750743866 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 5 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18665811419487 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.235725536942482 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7477920055389404 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7301965951919556 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032979730516672134 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1820008009672165 jet/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 )jet/accuracy_1_of_4‚0.978945791721344 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3239929676055908 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8739228844642639 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_4_of_4‚0.013899499550461769 /particle/accuracy_4_of_4‚0.9651159644126892 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936036467552185 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21022063493728638 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23545467853546143 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2816360294818878 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.67802494764328 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11653550714254379 +jet/accuracy_1_of_1‚0.33839666843414307 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884521961212158 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5923523306846619 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6300297975540161 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.472578763961792 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8106682300567627 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 epoch‚6 _step‚230 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7377114295959473 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923041462898254 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6639207005500793 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.396195650100708 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3513641953468323 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21167561411857605 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.848517119884491 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7164101600646973 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30749306082725525 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036878667771816254 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7759926319122314 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4679679870605469 +jet/accuracy_3_of_4‚0.06564100831747055 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928374290466309 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29250434041023254 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7646611332893372 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.052452486008405685 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039859410375356674 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21415381133556366 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8357504606246948 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992041826248169 'particle/f_score‚0.6333080530166626}XCîÆ=± À{ 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.767119288444519 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4736865758895874 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044114671647548676 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2194276601076126 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03627071902155876 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 jet/accuracy_4_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18723881244659424 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22371704876422882 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19921515882015228 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21661460399627686 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6565626859664917 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3778187930583954 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692741394042969 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6917603611946106 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8818972706794739 *jet/accuracy_1_of_2‚0.7985442876815796 *jet/accuracy_2_of_3‚0.5379233360290527 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17669321596622467 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19751830399036407 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_2_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6774737238883972 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9937629699707031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9016722440719604 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8818594813346863 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952654242515564 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7144115567207336 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8826342225074768 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6760363578796387 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6746310591697693 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6526148319244385 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23053103685379028 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951386451721191 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7800557017326355 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7651753425598145 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22398917376995087 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚2799 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6309691071510315 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0993061289191246 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6834532022476196 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_2_of_2‚0.14172162115573883 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7319905757904053 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.765900731086731 (particle/accuracy‚0.8433315753936768 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06356984376907349 particle/accuracy_5_of_5‚0 _runtime‚2398.046555724 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996243953704834 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7665168046951294 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7180811166763306 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6770930886268616 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5087500214576721 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.848517119884491 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6749030351638794 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03947584331035614 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48134884238243103 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9856293797492981 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6770930886268616 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.540668785572052 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03986300155520439 *jet/accuracy_1_of_3‚0.9283366203308105 particle/accuracy_3_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5887508988380432 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11271053552627563 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1706826388835907 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7993576526641846 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19351625442504883 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7663083672523499 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859439134597778 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23724229633808136 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03627071902155876 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9298042058944702 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.760002076625824 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.697464644908905 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9845884442329407 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1994244009256363 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.777563750743866 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 5 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18665811419487 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.235725536942482 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7477920055389404 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7301965951919556 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032979730516672134 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1820008009672165 jet/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 )jet/accuracy_1_of_4‚0.978945791721344 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3239929676055908 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8739228844642639 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_4_of_4‚0.013899499550461769 /particle/accuracy_4_of_4‚0.9651159644126892 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936036467552185 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21022063493728638 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23545467853546143 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2816360294818878 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.67802494764328 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11653550714254379 +jet/accuracy_1_of_1‚0.33839666843414307 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884521961212158 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5923523306846619 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6300297975540161 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.472578763961792 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8106682300567627 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 epoch‚6 _step‚230 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7377114295959473 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923041462898254 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6639207005500793 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.396195650100708 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3513641953468323 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21167561411857605 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.848517119884491 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7164101600646973 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30749306082725525 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036878667771816254 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7759926319122314 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4679679870605469 +jet/accuracy_3_of_4‚0.06564100831747055 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928374290466309 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29250434041023254 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7646611332893372 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.052452486008405685 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039859410375356674 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21415381133556366 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8357504606246948 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992041826248169 'particle/f_score‚0.6333080530166626 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6712158918380737 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4597400426864624 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.38250628113746643 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8078430891036987 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11333334445953369 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8388831615447998 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22398917376995087 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02735362946987152 particle/accuracy_4_of_5‚1 )loss/val_total_loss‚9.127449575498366 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04047080874443054 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5337244272232056 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7209983468055725 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_3‚0.05911188945174217 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13009274005889893 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21038818359375 % _timestamp‚1.7843599059705105e+09 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.399338036775589 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941222071647644 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475æ'‰?¾² ‚(:³ ã×ìÒŠú» disk.dm-0.out‚ 9694.49609375 disk.dm-1.out‚ 116.70703125$ memory_percent‚96.65887355804443 proc.memory.rssMB‚ 2883.1875 cpu‚26.16679725468037 proc.cpu.threads‚17 disk.dm-0.in‚ 4522.55078125 disk.rbd1.out‚ 5.70703125* proc.memory.percent‚17.597579956054688 network.recv‚671616 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.out‚ 9697.87109375) disk./.usagePercent‚39.50879486235069 disk.sda3.out‚ 116.70703125 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 network.sent‚ 4.363591e+06$ disk./.usageGB‚588.7095947265625" disk.nvme0n1.out‚ 9697.87109375 disk.sda.out‚ 116.70703125 disk.rbd1.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 4522.55078125! disk.nvme0n1.in‚ 4522.55078125( proc.memory.availableMB‚ 547.41015625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0LvÑœ³ ‚(:‘ ã×ìÒ̼ gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.153' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.230600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àf톑´ ‚(:† ò×ìÒࣨ¼ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëë(ѵ ‚(:Æ ò×ìÒÃõ¥½) disk./.usagePercent‚39.50879102223767# disk.nvme0c0n1.in‚ 4550.37890625 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 network.recv‚675956 disk.rbd1.in‚0$ disk.nvme0n1.out‚9721.7646484375 disk.sda2.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 743.3828125 disk.sda.in‚ 0.00390625 disk.drbd2165.out‚0 disk.dm-0.in‚ 4550.37890625 disk.sda3.out‚228.8125 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚228.8125 network.sent‚ 4.4209e+06$ disk./.usageGB‚588.7095375061035 disk.sda2.out‚0! disk.dm-0.out‚9721.7646484375" proc.memory.rssMB‚ 2716.7734375& disk.nvme0c0n1.out‚9721.7646484375 disk.drbd2165.in‚0) proc.memory.percent‚16.58186912536621 proc.cpu.threads‚17! disk.nvme0n1.in‚ 4550.37890625 disk.sda.out‚228.8125 disk.sda1.in‚0 disk.rbd1.out‚ 5.83203125 disk.rbd0.out‚0# memory_percent‚95.4627513885498 cpu‚26.16861123912194šRÇž¶ ‚(:“ ØìÒꧻ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{/¹Ñ· ‚(:Æ ØìÒÓØÕ½ disk.sda2.out‚0 cpu‚26.16961005549334 disk.rbd1.in‚0 disk.sda.in‚ 0.00390625# memory_percent‚95.4655647277832' disk.nvme0c0n1.out‚9971.83056640625 disk.nvme0n1.in‚ 4552.65625 disk.sda3.out‚ 229.203125 proc.cpu.threads‚17 network.recv‚679727) disk./.usagePercent‚39.50872420427124 disk.dm-1.out‚ 229.203125& proc.memory.availableMB‚ 742.921875% disk.nvme0n1.out‚9971.83056640625 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚ 5.8359375 disk.sda.out‚ 229.203125 disk.sda1.out‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2716.7734375 network.sent‚ 4.425875e+06$ disk./.usageGB‚588.7085418701172 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 4552.65625" disk.dm-0.out‚9971.83056640625) proc.memory.percent‚16.58186912536621 disk.nvme0c0n1.in‚ 4552.65625Oø4}Ev¸æ1¸ - _wandbruntime‚2439  _runtime‚2439QßÞɹ à ' lr-AdamW/pg1‚0.00041845692142728765 ' lr-AdamW/pg2‚0.00041845692142728765 trainer/global_step‚2849 % _timestamp‚1.7843599478217998e+09 _runtime‚2439.897105611 _step‚231²¾aκ È trainer/global_step‚2849 % _timestamp‚1.7843599478217998e+09 _runtime‚2439.897105611 _step‚231 ' lr-AdamW/pg1‚0.00041845692142728765 ' lr-AdamW/pg2‚0.00041845692142728765ç'lsÖ2» - _wandbruntime‚2439  _runtime‚2439›¡Ü'¼ ÖO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430951324110848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6132843 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3010109020866 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4996771 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98554782278144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2849 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3047475509760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98554885273088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚2439.901217615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402805 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3014708371456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26392828 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091979776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11129358 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091979776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402805 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3014367032320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433179741986816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3014708371456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433961092581824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0±*‹­á'½ ÛO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3009768470976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚436194450358272 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3010109020866 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4996771 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98554782278144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2849 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3047475509760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98554885273088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚2439.901217615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402805 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3014708371456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26392828 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091979776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11129358 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091979776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402805 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3014367032320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433179741986816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3014708371456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433961092581824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430951324110848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6132843 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3048502189056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3015379659264 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433326983016448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3015721005568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6134918 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431097834782720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436342704022016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98589338606592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524698 % _timestamp‚1.7843599484363353e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22607354 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433326983016448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431097851822080 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚436342704022016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4998516 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6134671 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98589237715968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11133434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433326965977088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3048584467456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22607356 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433326965977088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379659264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26411871 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26411871 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3048502189056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3015379659264 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433326983016448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3015721005568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6134918 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431097834782720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436342704022016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98589338606592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524698 % _timestamp‚1.7843599484363353e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22607354 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433326983016448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431097851822080 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚436342704022016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4998516 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6134671 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98589237715968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11133434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433326965977088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3048584467456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22607356 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433326965977088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379659264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269é`Nh2Á - _wandbruntime‚2440  _runtime‚2440¦– € 2loss/FBt1/assignment_loss‚0.013919702891132051 /loss/FRt2/detection_loss‚1.4885993475295227 _runtime‚2440.513189339 0loss/FRt1/assignment_loss‚1.9070968317666732 1loss/SRqqt2/detection_loss‚0.6739883696234615 /loss/FBt1/detection_loss‚0.6756102492011362 /loss/FBt2/detection_loss‚0.4655048185447561 $loss/total_loss‚8.84800341676663 epoch‚7 2loss/SRqqt2/assignment_loss‚0.5647214430164441 .loss/FRt1/detection_loss‚0.575414671470377 0loss/FRt2/assignment_loss‚0.8248650083027849 2loss/SRqqt1/assignment_loss‚0.6171477859797111 1loss/FBt2/assignment_loss‚0.09278748511041181 1loss/SRqqt1/detection_loss‚0.9483477033302188 trainer/global_step‚2849 % _timestamp‚1.7843599484380984e+09 _step‚234l~ìé à … 0loss/FRt2/assignment_loss‚0.8248650083027849 2loss/SRqqt1/assignment_loss‚0.6171477859797111 1loss/FBt2/assignment_loss‚0.09278748511041181 1loss/SRqqt1/detection_loss‚0.9483477033302188 trainer/global_step‚2849 % _timestamp‚1.7843599484380984e+09 _step‚234 2loss/FBt1/assignment_loss‚0.013919702891132051 /loss/FRt2/detection_loss‚1.4885993475295227 _runtime‚2440.513189339 0loss/FRt1/assignment_loss‚1.9070968317666732 1loss/SRqqt2/detection_loss‚0.6739883696234615 /loss/FBt1/detection_loss‚0.6756102492011362 /loss/FBt2/detection_loss‚0.4655048185447561 $loss/total_loss‚8.84800341676663 epoch‚7 2loss/SRqqt2/assignment_loss‚0.5647214430164441 .loss/FRt1/detection_loss‚0.575414671470377ê:C„‘Ä ‚(:† ØìÒ­¤—º gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N“ŸñØÅ ‚(:Í ØìҜӉ¼ disk.sda.in‚ 0.00390625 disk.sda1.in‚0" disk.dm-0.out‚9978.36474609375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚9978.36474609375( proc.memory.availableMB‚ 743.51171875) proc.memory.percent‚16.58186912536621 disk.sda2.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚9978.36474609375 disk.sda1.out‚0 disk.dm-1.out‚ 229.7734375 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625 disk.dm-0.in‚ 4552.65625$ memory_percent‚95.46196460723877 disk.nvme0n1.in‚ 4552.65625 disk.sda.out‚ 229.7734375 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚ 5.8984375" proc.memory.rssMB‚ 2716.7734375 network.recv‚682350) disk./.usagePercent‚39.50873879670069$ disk./.usageGB‚588.7087593078613 disk.sda3.out‚ 229.7734375 disk.rbd0.out‚0 cpu‚26.170841787467033 network.sent‚ 4.430001e+06 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 4552.65625ß‘9žÆ ‚(:“ ŸØìÒÓ°ñº gpu.0.gpu‚63.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚158.183' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.636599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0( ùÌÚÇ ‚(:Ï ŸØìҾͶ½ network.recv‚686977 disk.dm-1.out‚ 230.01953125 disk.nvme0n1.in‚ 4552.65625" proc.memory.rssMB‚ 2716.7734375) disk./.usagePercent‚39.50872778837672 disk.sda2.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4552.65625 network.sent‚ 4.492897e+06 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.drbd2165.out‚0 disk.dm-0.in‚ 4552.65625$ memory_percent‚95.46151161193848 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚ 5.8984375' proc.memory.availableMB‚ 743.5859375 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚9981.41748046875 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚9981.41748046875' disk.nvme0c0n1.out‚9981.41748046875) proc.memory.percent‚16.58186912536621$ disk./.usageGB‚588.7085952758789 disk.sda.out‚ 230.01953125 disk.sda3.out‚ 230.01953125 disk.drbd2165.in‚0 cpu‚26.171793357047445•ô ’È ‚(:‡ ®ØìÒ–šûº gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s{{aàÉ ‚(:Õ ®ØìÒ€›à½ network.sent‚ 4.49785e+06 disk.dm-0.in‚ 4611.76171875 disk.dm-1.in‚ 0.00390625 proc.cpu.threads‚17 disk.rbd1.in‚0% disk.nvme0c0n1.out‚9985.486328125! disk.nvme0n1.in‚ 4611.76171875) proc.memory.percent‚16.58186912536621 network.recv‚690170$ disk./.usageGB‚588.7085800170898 disk.dm-1.out‚ 230.25390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 230.25390625 disk.sda3.in‚ 0.00390625 disk.dm-0.out‚9985.486328125# disk.nvme0c0n1.in‚ 4611.76171875# disk.nvme0n1.out‚9985.486328125 disk.sda2.in‚0 disk.drbd2165.in‚0 cpu‚26.173255821560552) disk./.usagePercent‚39.50872676434658 disk.sda.in‚ 0.00390625 disk.sda1.out‚0$ memory_percent‚95.46151161193848 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 230.25390625 disk.rbd1.out‚ 5.98486328125 disk.drbd2165.out‚0' proc.memory.availableMB‚ 743.5859375" proc.memory.rssMB‚ 2716.7734375Ë]Ž2Ê - _wandbruntime‚2480  _runtime‚24800¤ðîÆË À _runtime‚2480.374104538 _step‚235 & lr-AdamW/pg1‚0.0004177836768314104 & lr-AdamW/pg2‚0.0004177836768314104 trainer/global_step‚2899 $ _timestamp‚1.784359988298753e+09b ¬ËÌ Å & lr-AdamW/pg1‚0.0004177836768314104 & lr-AdamW/pg2‚0.0004177836768314104 trainer/global_step‚2899 $ _timestamp‚1.784359988298753e+09 _runtime‚2480.374104538 _step‚235ë0œä2Í - _wandbruntime‚2480  _runtime‚2480ê-AÞ'Î ØO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438276857704448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3065403873280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100277674667008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚2899 % _timestamp‚1.7843599882998428e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26857257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443606838963200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441337600251582 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11320630 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3060725507774 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838963200 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440541776429056 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11320349 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26847280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100277565380608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3098820685824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443607197341696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438276874743808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚440541793468416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6236609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3065403873280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26857257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5084021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5084018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443607197341696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3065062534144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441337242662332 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888WOÑ|.Ï ÝO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440541776429056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3065062534144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441337242662332 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438276857704448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3065403873280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100277674667008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚2899 % _timestamp‚1.7843599882998428e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26857257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443606838963200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441337600251582 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11320630 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3060725507774 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838963200 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440541776429056 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11320349 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26847280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100277565380608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3098820685824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443607197341696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438276874743808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚440541793468416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6236609 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3065403873280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMono«µitor.on_train_batch_start/active.small_pool.allocated‚26857257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5084021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5084018 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443607197341696 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22988956 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22988954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97212173453312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100312122253824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011606 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843599889148211e+09 _runtime‚2480.990473358 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26866323 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440689017458688 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416507392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11324101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3066075161088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441484764387926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6238585 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3066075161088Þ˜eLí&Ò çM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26866323 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440689017458688 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416507392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11324101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3066075161088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441484764387926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6238585 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3066075161088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22988956 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22988954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97212173453312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100312122253824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011606 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843599889148211e+09 _runtime‚2480.990473358í»ÃŽ®2Ó - _wandbruntime‚2480  _runtime‚2480I™È»Ô € 2loss/SRqqt2/assignment_loss‚0.5309736164662461 /loss/FRt1/detection_loss‚0.5638283518972815 1loss/SRqqt2/detection_loss‚0.6564798563871941 2loss/SRqqt1/assignment_loss‚0.6705279791185974 1loss/FBt2/assignment_loss‚0.06299911787229484 /loss/FBt1/detection_loss‚1.1827668126982946 trainer/global_step‚2899 _runtime‚2480.991780265 0loss/FRt1/assignment_loss‚1.8787567681866915 /loss/FRt2/detection_loss‚1.4484477706703058 %loss/total_loss‚9.438530891826783 epoch‚7 _step‚238 0loss/FRt2/assignment_loss‚0.7756887009457683 0loss/FBt1/assignment_loss‚0.1066806451467525 1loss/SRqqt1/detection_loss‚1.0459797362017693 /loss/FBt2/detection_loss‚0.5154015362355858 % _timestamp‚1.7843599889164498e+09Û1­ Õ … 2loss/SRqqt2/assignment_loss‚0.5309736164662461 /loss/FRt1/detection_loss‚0.5638283518972815 1loss/SRqqt2/detection_loss‚0.6564798563871941 2loss/SRqqt1/assignment_loss‚0.6705279791185974 1loss/FBt2/assignment_loss‚0.06299911787229484 /loss/FBt1/detection_loss‚1.1827668126982946 trainer/global_step‚2899 _runtime‚2480.991780265 0loss/FRt1/assignment_loss‚1.8787567681866915 /loss/FRt2/detection_loss‚1.4484477706703058 %loss/total_loss‚9.438530891826783 epoch‚7 _step‚238 0loss/FRt2/assignment_loss‚0.7756887009457683 0loss/FBt1/assignment_loss‚0.1066806451467525 1loss/SRqqt1/detection_loss‚1.0459797362017693 /loss/FBt2/detection_loss‚0.5154015362355858 % _timestamp‚1.7843599889164498e+09îýDð?ëÖ ‚(:à ½ØìÒãÙѼ network.sent‚ 4.502817e+06 disk.sda1.out‚0 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚89.36376953125$ memory_percent‚95.46191692352295 network.recv‚693462) disk./.usagePercent‚39.52593968691876 disk.dm-0.in‚ 4612.44140625 disk.sda.out‚ 230.6015625! disk.dm-0.out‚9996.1279296875 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9996.1279296875) proc.memory.percent‚16.58186912536621 cpu‚26.17433417941481 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 230.6015625" proc.memory.rssMB‚ 2716.7734375 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625$ disk./.usageGB‚588.9650650024414 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4612.44140625 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0n1.out‚9996.1279296875( proc.memory.availableMB‚ 743.51953125 disk.drbd2165.out‚0 disk.dm-1.out‚ 230.6015625! disk.nvme0n1.in‚ 4612.44140625ékΤ’× ‚(:‡ ½ØìÒö¦¨½ gpu.0.gpu‚41.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÓ“Ø ‚(:ˆ ÌØìÒü×κ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚196.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.3806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0( ÀñÙ ‚(:æ ÌØìÒ™‰Ž½ disk.dm-0.in‚ 4612.44140625# disk.dm-0.out‚10249.68603515625 network.recv‚697833 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0 proc.cpu.threads‚17$ proc.memory.availableMB‚743.8125 disk.dm-1.out‚ 231.14453125 network.sent‚ 4.565915e+06 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0( disk.nvme0c0n1.out‚10249.68603515625$ memory_percent‚95.46012878417969 cpu‚26.175859960421594 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 231.14453125# disk.nvme0c0n1.in‚ 4612.44140625) proc.memory.percent‚16.58186912536621 disk.sda.out‚ 231.14453125" proc.memory.rssMB‚ 2716.7734375) disk./.usagePercent‚39.52594839117493$ disk./.usageGB‚588.9651947021484 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚89.77783203125! disk.nvme0n1.in‚ 4612.44140625 disk.rbd0.in‚0& disk.nvme0n1.out‚10249.68603515625ÅÑÚ ‚(:’ ÛØìÒ¯¼Œ¼ gpu.0.gpu‚26.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï^SÜõÛ ‚(:ê ÛØìÒו´½ disk.drbd2165.out‚0 disk.dm-1.out‚ 231.33203125$ memory_percent‚95.46163082122803 network.recv‚701220# disk.nvme0c0n1.in‚ 4612.44140625 disk.sda.out‚ 231.33203125 disk.rbd1.in‚ 0.01953125& disk.nvme0n1.out‚10258.73291015625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 743.56640625 cpu‚26.177275100095176$ disk./.usageGB‚588.9649124145508 disk.sda.in‚ 0.00390625 disk.dm-0.in‚ 4612.44140625 disk.dm-1.in‚ 0.00390625) proc.memory.percent‚16.58186912536621# disk.dm-0.out‚10258.73291015625 disk.rbd0.in‚0! disk.nvme0n1.in‚ 4612.44140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) disk./.usagePercent‚39.52592944661739 disk.sda3.out‚ 231.33203125" proc.memory.rssMB‚ 2716.7734375 network.sent‚ 4.570887e+06 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚89.86376953125( disk.nvme0c0n1.out‚10258.73291015625 disk.sda3.in‚ 0.00390625(082Ü - _wandbruntime‚2520  _runtime‚2520"7÷ÒÁÝ » trainer/global_step‚2949  _timestamp‚1.7843600286e+09 _runtime‚2520.680221071 _step‚239 & lr-AdamW/pg1‚0.0004170975683820385 & lr-AdamW/pg2‚0.0004170975683820385uH)éÆÞ À _runtime‚2520.680221071 _step‚239 & lr-AdamW/pg1‚0.0004170975683820385 & lr-AdamW/pg2‚0.0004170975683820385 trainer/global_step‚2949  _timestamp‚1.7843600286e+09ï\ö©r2ß - _wandbruntime‚2520  _runtime‚2520•jqžà ØO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451019944325120 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445602391298048 trainer/global_step‚2949 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚445602408337408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102000387389440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447903844950016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3116099375104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362922 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98850201233408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332090 O?DeviceStatsMonitor.on_train_batch_stγ3‰Ùart/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447903827910656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3116099375104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11511248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102000271811584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3150186156032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5171271 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362924 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362922 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3115758035968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98850182621184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585946624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843600286007888e+09 _step‚240 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362924 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448713392742840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451019944325120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3111001444792Éõvã'á ÝO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3115758035968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6340243 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27311709 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27311709 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27301732 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451019585946624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3111341994682 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585946624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843600286007888e+09 _step‚240 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362924 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448713392742840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451019944325120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3111001444792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451019944325120 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445602391298048 trainer/global_step‚2949 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚445602408337408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102000387389440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447903844950016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3116099375104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362922 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98850201233408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332090 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447903827910656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3116099375104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11511248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102000271811584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3150186156032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5171271 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362924 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362922 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3115758035968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98850182621184 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843600292146719e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448051085979648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451168197988864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102034725447168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448861272057796 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451167839603200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3151202637312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6342012 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112353048516 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27320775 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112009216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98883611379712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691331 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448051085979648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚2521.290322829 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445748901969920 trainer/global_step‚2949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3116770662912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11515025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102034832629248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27320775 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ê¯NÒä çM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6342284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11515025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102034832629248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27320775 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3151114067456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11515300 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448051068940288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843600292146719e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448051085979648 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451168197988864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102034725447168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448861272057796 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451167839603200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3151202637312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6342012 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚LÅæ  3112353048516 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27320775 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112009216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98883611379712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691331 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448051085979648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚2521.290322829 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445748901969920 trainer/global_step‚2949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3116770662912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 trainer/global_step‚2999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458432691308544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481590 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103722951988224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452927924891648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3161617931700 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843600691858554e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332930048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23744524 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27766161 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3201215862784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510685 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456089900412598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455265879392256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794876928 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6443481 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11701999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100521639159808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270ˆ©€$å'ñ ßO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455265879392256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3166453537792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27756184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456089542823348 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500706 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23744522 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100521620547584 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27756184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27766161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3166794876928 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332930048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23744524 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27766161 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3201215862784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510685 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456089900412598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455265879392256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794876928 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6443481 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11701999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100521639159808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201312828416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 trainer/global_step‚2999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458432691308544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481590 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103722951988224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452927924891648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3161617931700 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843600691858554e+09 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5260266 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3162969535424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2999 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3167466164736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27765236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466164736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103757394438656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3202235853312 % _timestamp‚1.7843600697990725e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3162628985422 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23752154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3167807511040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458580586586624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5260263 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100555049306112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517390 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100555067918336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775227 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11705763 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944972288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458580944972288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088ôåhöï&ô éM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23752156 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27775227 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455413137461248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455413120421888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103757285159424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453074452602880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27765236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466164736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103757394438656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3202235853312 % _timestamp‚1.7843600697990725e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3162628985422 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23752154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3167807511040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458580586586624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5260263 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100555049306112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517390 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100555067918336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775227 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11705763 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944972288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458580944972288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11706028 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6445500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚2561.874651018 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5260266 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3162969535424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2999 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3167466164736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3217490378752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460253475524608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079913472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462627947913216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465845079913472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5345768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460253458485248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3217149039616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3212574968498 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24126122 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24126122 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252427496960 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52346737 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24126124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462627930873856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463465692903856 % _timestamp‚1.7843601097195685e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52336758 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462627930873856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212234418608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465845438291968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28210636 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3252513976832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28210636 trainer/global_step‚3049 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28220613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102193077086208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3217490378752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚463466050493106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3217149039616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6547160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5345771 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105445591063040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.aB®Âellocated‚462627947913216 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24126122 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252427496960 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52346737 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24126124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462627930873856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463465692903856 % _timestamp‚1.7843601097195685e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52336758 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462627930873856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3212234418608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465845438291968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28210636 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3252513976832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28210636 trainer/global_step‚3049 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28220613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102193077086208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3217490378752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚463466050493106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3217149039616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6547160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5345771 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105445591063040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462627947913216 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3217490378752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460253475524608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079913472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462627947913216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465845079913472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5345768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460253458485248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3217149039616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3212574968498øæÀ¹ú2„ - _wandbruntime‚2602  _runtime‚2602å‘Ååé&… ãM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465993691955712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105480045811712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333570048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28219688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3213586022332 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105479938629632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353442 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11896450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3218161666560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463613214629450 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465993691955712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3218161666560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548937 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253539966976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24133754 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28229679 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3218503012864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463613572218812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28219688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚3049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462775188942848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213245472330 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3218503012864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976…^„uî&† èM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462775188942848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213245472330 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3218503012864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465993691955712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105480045811712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333570048 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28219688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3213586022332 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105479938629632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353442 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11896450 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3218161666560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463613214629450 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465993691955712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3218161666560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548937 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253539966976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24133754 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28229679 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3218503012864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463613572218812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28219688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚3049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ù’· °2‡ - _wandbruntime‚2602  _runtime‚2602åDHˆ ÿ 2loss/SRqqt1/assignment_loss‚0.6126016814269633 1loss/FBt1/assignment_loss‚0.06683039045901751 2loss/FBt2/assignment_loss‚0.032681202745219046 /loss/FRt1/detection_loss‚0.5689376910608641 /loss/FRt2/detection_loss‚1.4803565573517015 trainer/global_step‚3049 1loss/SRqqt2/detection_loss‚0.5980489643386658 /loss/FBt1/detection_loss‚1.2713074351195246 /loss/FBt2/detection_loss‚0.5418860837887747 $loss/total_loss‚9.49915752536727 epoch‚7 _runtime‚2602.410123809 0loss/FRt2/assignment_loss‚0.7127126279999229 /loss/SRqqt1/detection_loss‚1.13794449618442 0loss/FRt1/assignment_loss‚1.9332333269992474 2loss/SRqqt2/assignment_loss‚0.5426170678929485 % _timestamp‚1.7843601103350394e+09 _step‚250¾õ ˆ ‰ „ /loss/FRt1/detection_loss‚0.5689376910608641 /loss/FRt2/detection_loss‚1.4803565573517015 trainer/global_step‚3049 1loss/SRqqt2/detection_loss‚0.5980489643386658 /loss/FBt1/detection_loss‚1.2713074351195246 /loss/FBt2/detection_loss‚0.5418860837887747 $loss/total_loss‚9.49915752536727 epoch‚7 _runtime‚2602.410123809 0loss/FRt2/assignment_loss‚0.7127126279999229 /loss/SRqqt1/detection_loss‚1.13794449618442 0loss/FRt1/assignment_loss‚1.9332333269992474 2loss/SRqqt2/assignment_loss‚0.5426170678929485 % _timestamp‚1.7843601103350394e+09 _step‚250 2loss/SRqqt1/assignment_loss‚0.6126016814269633 1loss/FBt1/assignment_loss‚0.06683039045901751 2loss/FBt2/assignment_loss‚0.032681202745219046úŒêᜊ ‚(:‘ µÙì҃读 gpu.0.gpu‚22.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.571999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è ]ÚŠ‹ ‚(:Þ µÙìÒáòî¼ disk.sda.out‚ 233.765625 disk.sda3.in‚ 0.00390625 proc.cpu.threads‚17 disk.drbd2165.out‚0) proc.memory.percent‚16.58186912536621 disk.drbd2165.in‚0 disk.dm-1.out‚ 233.765625 disk.rbd1.out‚ 90.283203125" proc.memory.rssMB‚ 2716.7734375 network.sent‚ 4.716883e+06$ disk./.usageGB‚588.9634857177734" disk.dm-0.out‚10289.1s½¦}_025390625 disk.dm-1.in‚ 0.00390625( proc.memory.availableMB‚ 498.34765625 cpu‚26.183849758867076 disk.sda1.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 4717.97265625! disk.nvme0n1.in‚ 4717.97265625 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 233.765625 network.recv‚723609 disk.dm-0.in‚ 4717.97265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01953125" memory_percent‚96.958327293396* disk./.usagePercent‚39.525833699799584 disk.sda1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚10289.0986328125% disk.nvme0n1.out‚10289.1025390625M«¸šàŒ ‚(:Õ ÄÙìÒÖæÔ¼ disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625) proc.memory.percent‚16.58186912536621 disk.dm-1.in‚ 0.00390625 disk.sda2.in‚0 disk.drbd2165.in‚0# memory_percent‚96.9585657119751) disk./.usagePercent‚39.52584138002561# disk.nvme0c0n1.in‚ 4717.97265625 disk.dm-0.out‚ 10292.859375 cpu‚26.184794879349177 network.sent‚ 4.779764e+06 disk.dm-1.out‚ 234.0859375 disk.rbd1.out‚ 90.345703125 disk.sda.out‚ 234.0859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.recv‚728196 disk.sda3.out‚ 234.0859375 disk.dm-0.in‚ 4717.97265625 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 10292.859375! disk.nvme0n1.in‚ 4717.97265625 disk.sda.in‚ 0.00390625! disk.nvme0n1.out‚ 10292.859375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 498.30859375 proc.cpu.threads‚17$ disk./.usageGB‚588.9636001586914" proc.memory.rssMB‚ 2716.7734375=ñk“ ‚(:ˆ ÄÙìÒ™›â¼ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g}£Ö“Ž ‚(:ˆ ÓÙìҀظº gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08í/nï ‚(:ä ÓÙìÒÎù÷¼ disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 4717.97265625 disk.sda.in‚ 0.00390625 disk.sda1.out‚0" proc.memory.rssMB‚ 2716.7734375 cpu‚26.186282255686795 disk.rbd1.out‚ 90.345703125! disk.nvme0n1.in‚ 4717.97265625 network.recv‚731381 disk.dm-1.out‚ 234.61328125 disk.sda3.out‚ 234.61328125) proc.memory.percent‚16.58186912536621$ disk./.usageGB‚588.9643859863281 disk.dm-0.in‚ 4717.97265625 disk.rbd0.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 498.4609375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 234.61328125 network.sent‚ 4.784787e+06* disk./.usagePercent‚39.525894117577664 disk.sda2.in‚0 disk.rbd1.in‚ 0.01953125 disk.sda3.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625' disk.nvme0c0n1.out‚10300.2880859375% disk.nvme0n1.out‚10300.2880859375 disk.drbd2165.in‚0" disk.dm-0.out‚10300.2880859375# memory_percent‚96.9576358795166 proc.cpu.threads‚17«k`D2 - _wandbruntime‚2642  _runtime‚2642b²º{Ç‘ Á & lr-AdamW/pg2‚0.0004149625047016619 trainer/global_step‚3099 % _timestamp‚1.7843601500117185e+09 _runtime‚2642.087082737 _step‚251 & lr-AdamW/pg1‚0.0004149625047016619_IóÌ’ Æ _step‚251 & lr-AdamW/pg1‚0.0004149625047016619 & lr-AdamW/pg2‚0.0004149625047016619 trainer/global_step‚3099 % _timestamp‚1.7843601500117185e+09 _runtime‚2642.087082737ûßù2“ - _wandbruntime‚2642  _runtime‚2642guüÞ'” ØO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3268185880576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3267844541440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473258185275392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24507724 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3099 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3268185880576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28665088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28675065 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103864515012608 _step‚252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24507722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470842200573614 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3262850905516 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473258185275392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469989999394816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5433018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103864496400384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467578992078848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469989982355456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6651122 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168175639040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467579009118208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28675065 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826896896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚469989982355456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3267844541440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3303660626432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28665088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5433021 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469989999394816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53172810 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182789 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12084399 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚2642.0883327675è[‚ã'• ÝO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473258185275392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24507724 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3099 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3268185880576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28665088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28675065 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103864515012608 _step‚252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24507722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470842200573614 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3262850905516 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473258185275392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469989999394816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5433018 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103864496400384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467578992078848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469989982355456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6651122 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168175639040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467579009118208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984364 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28675065 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826896896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚469989982355456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3267844541440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3303660626432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28665088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5433021 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469989999394816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53172810 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182789 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12084399 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚2642.088332767 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3268185880576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3267844541440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269üC…RÑ2– - _wandbruntime‚2642  _runtime‚2642zâ%Žg — äM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚2642.702661507 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198514688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473406438939136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470137223385088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12088166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3268857168384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3304604710400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24515354 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107202637051392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959238 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6653400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107202529869312 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12087903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470989722299320 % _timestamp‚1.7843601506254752e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189494 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470989364709958 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28684131 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 N>DeviceStatsMonitor.on_tê=Üуrain_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467725519790080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264202509240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198514688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6653140 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚467725502750720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28674140 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269OÍO½ï&˜ éM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28684131 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12087903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470989722299320 % _timestamp‚1.7843601506254752e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189494 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470989364709958 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28684131 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467725519790080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264202509240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198514688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6653140 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚467725502750720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28674140 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚2642.702661507 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198514688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473406438939136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470137223385088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12088166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3268857168384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3304604710400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24515354 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107202637051392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959238 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6653400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107202529869312 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚2682.612676706 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477352050876416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29129517 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚477352050876416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480670932258816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5520268 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54008862 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29119540 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108890840013312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105535934326784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480670573880320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478217993064872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3318881382400 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6754619 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3313467392424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573880320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480670932258816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018841 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29129517 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105535952939008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24889322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108890726532608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474904525672448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313807942314 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29119540 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29119540 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108890840013312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105535934326784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480670573880320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478217993064872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3318881382400 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6754619 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3313467392424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573880320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480670932258816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bMEÔytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018841 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29129517 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105535952939008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24889322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108890726532608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474904525672448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313807942314 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29119540 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚2682.612676706 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477352050876416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29129517 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚477352050876416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888€”®=*2¦ - _wandbruntime‚2683  _runtime‚2683Ÿ>é&§ ãM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12278414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24896956 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6756676 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12278692 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3319894016512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480819185922560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚3149 _step‚257 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3319552670208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3355912429568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379828 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3314478446146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚ 2683.22823635 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025546 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚477499291906048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925170167808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996148 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105569363085312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24896954 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29128592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29138583 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29138583 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477499274866688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478365514790466 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3319552670208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105569381697536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480818827536896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24896954 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475051053383680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480819185922560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54025546 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0*»¹î&¨ èM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24896954 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475051053383680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480819185922560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54025546 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12278414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24896956 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6756676 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12278692 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3319894016512 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480819185922560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚3149 _step‚257 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3319552670208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3355912429568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379828 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3314478446146 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚ 2683.22823635 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025546 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚477499291906048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925170167808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996148 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105569363085312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24896954 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29128592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29138583 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29138583 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477499274866688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478365514790466 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3319552670208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105569381697536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480818827536896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278q°‘2© - _wandbruntime‚2683  _runtime‚2683Ù ¢_ª  %loss/total_loss‚8.878163297411156 /loss/FBt2/detection_loss‚0.4578258614637889 epoch‚7 0loss/FRt2/assignment_loss‚0.7012170201764917 2loss/FBt2/assignment_loss‚0.041585229754113495 /loss/FRt2/detection_loss‚1.6211662190628044 1loss/SRqqt2/detection_loss‚0.6826172615947045 trainer/global_step‚3149 $ _timestamp‚1.784360191154915e+09 0loss/FRt1/assignment_loss‚1.8438511155070454 1loss/FBt1/assignment_loss‚0.10424755403373638 /loss/FRt1/detection_loss‚0.5930983292161448 _runtime‚2683.229990531 _step‚258 2loss/SRqqt1/assignment_loss‚0.6589638876092485 2loss/SRqqt2/assignment_loss‚0.6053707774489183 1loss/SRqqt1/detection_loss‚1.0177701409356517 /loss/FBt1/detection_loss‚0.5504499006085097ñÍ¥/ « † /loss/FRt1/detection_loss‚0.5930983292161448 _runtime‚2683.229990531 _step‚258 2loss/SRqqt1/assignment_loss‚0.6589638876092485 2loss/SRqqt2/assignment_loss‚0.6053707774489183 1loss/SRqqt1/detection_loss‚1.0177701409356517 /loss/FBt1/detection_loss‚0.5504499006085097 %loss/total_loss‚8.878163297411156 /loss/FBt2/detection_loss‚0.4578258614637889 epoch‚7 0loss/FRt2/assignment_loss‚0.7012170201764917 2loss/FBt2/assignment_loss‚0.041585229754113495 /loss/FRt2/detection_loss‚1.6211662190628044 1loss/SRqqt2/detection_loss‚0.6826172615947045 trainer/global_step‚3149 $ _timestamp‚1.784360191154915e+09 0loss/FRt1/assignment_loss‚1.8438511155070454 1loss/FBt1/assignment_loss‚0.10424755403373638‚$v/h“¬ ‚(:ˆ €ÚìÒÓ¨ç» gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãsͱê­ ‚(:ß €ÚìÒ¸ùè¼ disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 4783.80859375 disk.rbd0.out‚0$ disk./.usageGB‚588.9645957946777 disk.sda3.in‚ 0.00390625 disk.drbd2165.out‚0$ memory_percent‚96.95923328399658" proc.memory.rssMB‚ 2716.7734375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 235.734375' disk.nvme0c0n1.out‚10312.5029296875 disk.sda.out‚ 235.734375 disk.sda3.out‚ 235.734375 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 4.857685e+06 network.recv‚743263" disk.dm-0.out‚10312.5029296875 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125( proc.memory.availableMB‚ 498.19921875! disk.nvme0n1.in‚ 4783.80859375% disk.nvme0n1.out‚10312.5029296875 disk.rbd1.out‚ 90.486328125) proc.memory.percent‚16.58186912536621 cpu‚26.189096449093732) disk./.usagePercent‚39.52590819799205 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 4783.80859375 disk.sda.in‚ 0.00390625½’6® ‚(:’ ÚìÒïÞܺ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ·n­ë¯ ‚(:à ÚìÒˆ¾í¼$ disk./.usageGB‚588.9644546508789 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 4783.80859375 proc.cpu.threads‚17 disk.sda1.in‚0( disk.nvme0c0n1.out‚10317.48388671875( proc.memory.availableMB‚ 497.19140625 cpu‚26.18814594115691 disk.sda3.out‚ 236.140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01953125 disk.rbd0.out‚0) proc.memory.percent‚16.58186912536621 network.recv‚747793 disk.sda.in‚ 0.00390625 disk.dm-1.out‚ 236.140625" proc.memory.rssMB‚ 2716.7734375 disk.sda3.in‚ 0.00390625# memory_percent‚96.9653844833374 network.sent‚ 4.920406e+06& disk.nvme0n1.out‚10317.48388671875 disk.rbd1.out‚ 90.486328125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 4783.80859375) disk./.usagePercent‚39.52589872571328! disk.nvme0n1.in‚ 4783.80859375 disk.sda.out‚ 236.140625 disk.sda1.out‚0# disk.dm-0.out‚10317.48388671875 disk.dm-1.in‚ 0.00390625‘ÔQ“° ‚(:ˆ žÚìÒ‰äÒº gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.652' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VWúÃã± ‚(:Ø žÚìÒ¾×æ¼ disk.drbd2165.in‚0 disk.dm-0.out‚ 10320.6640625 disk.dm-1.out‚ 236.58203125" disk.nvme0c0n1.in‚ 4842.5703125' proc.memory.availableMB‚ 497.4140625 network.sent‚ 4.925377e+06 network.recv‚750895 disk.dm-0.in‚ 4842.5703125$ memory_percent‚96.96402549743652 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 10320.6640625) proc.memory.percent‚16.58186912536621 disk.sda3.out‚ 236.58203125$ disk./.usageGB‚588.9645843505859 disk.rbd1.in‚ 0.01953125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 236.58203125" proc.memory.rssMB‚ 2716.7734375 cpu‚26.187664792215752 disk.sda.in‚ 0.00390625) disk./.usagePercent‚39.52590742996945 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚ 90.564453125" disk.nvme0n1.out‚ 10320.6640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4842.5703125*BÏ2² - _wandbruntime‚2723  _runtime‚2723«dËbƳ À trainer/global_step‚3199 $ _timestamp‚1.784360231242718e+09 _runtime‚2723.323268874 _step‚259 & lr-AdamW/pg1‚0.0004134756394517494 & lr-AdamW/pg2‚0.0004134756394517494¼Ññ“Ë´ Å _step‚259 & lr-AdamW/pg1‚0.0004134756394517494 & lr-AdamW/pg2‚0.0004134756394517494 trainer/global_step‚3199 $ _timestamp‚1.784360231242718e+09 _runtime‚2723.323268874ƒëë(Õ2µ - _wandbruntime‚2723  _runtime‚2723ýö;gð¶ ÚO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25270922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3369576884224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12465841 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482230059266048 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5607518 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚3199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320863744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25270924 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29583969 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12465578 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843602312487671e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚2723.324312785 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488083679242240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484714102358016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3364083879332 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ó< å'· ßO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3369576884224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12465841 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482230059266048 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5607518 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚3199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320863744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25270924 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29583969 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12465578 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7843602312487671e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚2723.324312785 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488083679242240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484714102358016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3364083879332 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25270922„id«2¸ - _wandbruntime‚2723  _runtime‚2723”ÎCé&¹ ãM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107240819623936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚261 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482376586977280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12469656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚ 2723.93689579 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484861326348288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485741664870974 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25278556 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3370248172032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6860126 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843602318613214e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488231574520320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3407012271104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3365435483056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29583044 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932905984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370248172032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485742022460336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25278554 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5609263 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25278554 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29583044 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3370248172032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6860126 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843602318613214e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488231574520320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3407012271104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3365435483056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29583044 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932905984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370248172032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485742022460336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25278554 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_poÅ3qaol.freed‚5609263 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25278554 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29583044 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484861326348288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485741664870974 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591…hýá2» - _wandbruntime‚2723  _runtime‚2723ZŠg̼ ÿ _runtime‚2723.938954718 0loss/FRt1/assignment_loss‚1.8188319652082652 2loss/SRqqt1/assignment_loss‚0.7122536762667364 2loss/SRqqt2/assignment_loss‚0.5219797230181784 1loss/FBt1/assignment_loss‚0.07853400200146154 /loss/FRt1/detection_loss‚0.5378203222984168 1loss/SRqqt2/detection_loss‚0.6814732888865058 /loss/FBt1/detection_loss‚0.5334252202228527 1loss/FBt2/assignment_loss‚0.03370944285059185 1loss/SRqqt1/detection_loss‚0.9599092267389882 /loss/FBt2/detection_loss‚0.5234162715405546 trainer/global_step‚3199 $ _timestamp‚1.784360231863804e+09 $loss/total_loss‚8.73080325648571 0loss/FRt2/assignment_loss‚0.7422252295853256 _step‚262 /loss/FRt2/detection_loss‚1.5872248878678332 epoch‚7”Z±» ½ „ 1loss/FBt2/assignment_loss‚0.03370944285059185 1loss/SRqqt1/detection_loss‚0.9599092267389882 /loss/FBt2/detection_loss‚0.5234162715405546 trainer/global_step‚3199 $ _timestamp‚1.784360231863804e+09 $loss/total_loss‚8.73080325648571 0loss/FRt2/assignment_loss‚0.7422252295853256 _step‚262 /loss/FRt2/detection_loss‚1.5872248878678332 epoch‚7 _runtime‚2723.938954718 0loss/FRt1/assignment_loss‚1.8188319652082652 2loss/SRqqt1/assignment_loss‚0.7122536762667364 2loss/SRqqt2/assignment_loss‚0.5219797230181784 1loss/FBt1/assignment_loss‚0.07853400200146154 /loss/FRt1/detection_loss‚0.5378203222984168 1loss/SRqqt2/detection_loss‚0.6814732888865058 /loss/FBt1/detection_loss‚0.5334252202228527†¡f”ž¾ ‚(:“ ­ÚìÒ´øå¼ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅŸºð¿ ‚(:å ­ÚìÒ‘Á™¾ disk.sda3.in‚ 0.00390625 proc.cpu.threads‚17 network.sent‚ 4.930403e+06" disk.nvme0c0n1.in‚ 4842.5703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.744542121887207 network.recv‚754433 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625$ memory_percent‚97.11053371429443( proc.memory.availableMB‚ 473.41015625 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚ 90.564453125( disk.nvme0c0n1.out‚10325.34326171875& disk.nvme0n1.out‚10325.34326171875 disk.dm-0.in‚ 4842.5703125$ disk./.usageGB‚588.9644737243652# disk.dm-0.out‚10325.34326171875 disk.nvme0n1.in‚ 4842.5703125 disk.sda.out‚ 236.8984375 disk.sda3.out‚ 236.8984375 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 236.8984375# proc.memory.rssMB‚ 2743.42578125 cpu‚26.191788572793843* disk./.usagePercent‚39.525900005750955 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.00390625ÂYÅ2À - _wandbruntime‚2742  _runtime‚2742Fä@èò=Á ì{ =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6675694584846497 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2347719520330429 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14124062657356262 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2012448012828827 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8640019297599792 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929137229919434 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5648747682571411 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04423118755221367 -"Purity/2FRt*SRqqt1FBt/event_purity‚0.1875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5593847036361694 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06653472781181335 *jet/accuracy_3_of_4‚0.0816865786910057 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47300925850868225 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 +jet/accuracy_3_of_3‚0.07437843829393387 +jet/accuracy_4_of_4‚0.01628045178949833 'particle/f_score‚0.6453225612640381 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6167340278625488 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23497061431407928 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9938890337944031 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4322916865348816 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12813428044319153 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30262982845306396 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22964322566986084 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_3_of_4‚0.9938492774963379 epoch‚7 (particle/accuracy‚0.8499077558517456 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5512500405311584 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9896675944328308 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6606543064117432 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.039802540093660355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9965478777885437 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9344983696937561 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6804207563400269 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚8.890356529686532 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872538447380066 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3953187167644501 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4798799157142639 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4597752094268799 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6467170119285583 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *particle/specificity‚0.908441424369812 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7346331477165222 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19811661541461945 /particle/accuracy_4_of_4‚0.9555649757385254 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24517996609210968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19737409055233002 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0830746665596962 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20499177277088165 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9926759600639343 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.899092972278595 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6659560203552246 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18791702389717102 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.043386880308389664 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.786823570728302 /particle/accuracy_2_of_2‚0.9998962879180908 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24697645008563995 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7827073335647583 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9076341986656189 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04524856060743332 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20878787338733673 *jet/accuracy_2_of_3‚0.5651280879974365 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2954440712928772 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6864126324653625 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.669317901134491 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8548216223716736 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7720421552658081 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1908043920993805 *jet/accuracy_1_of_1‚0.3394206464290619 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7149572968482971 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4471101760864258 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6582632660865784 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9867425560951233 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8811541795730591 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.64238440990448 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_2‚0.1519448310136795 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 % _timestamp‚1.7843602507932901e+09 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24750939011573792 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12395044416189194 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2187572866678238 6validation_average_jet_accuracy‚0.3818936049938202 trainer/global_step‚3199 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7182226777076721 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7705357670783997 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7621236443519592 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8181013464927673 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961658716201782 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22776247560977936 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0451769083738327 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.740471363067627 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7137414216995239 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44890880584716797 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 /particle/accuracy_3_of_3‚0.9994933605194092 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8739606738090515 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7343984246253967 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.568744957447052 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6268050074577332 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11105038225650787 particle/accuracy_1_of_1‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (jet/accuracy_1_of_4‚0.97369384765625 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29549145698547363 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17836639285087585 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.034710388630628586 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24989673495292664 jet/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6649784445762634 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952138066291809 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3933490514755249 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.554885745048523 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.039802540093660355 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _runtime‚2742.869443121 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6738050580024719 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940407276153564 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7972599864006042 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6255766749382019 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7113804221153259 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 *jet/accuracy_2_of_4‚0.8177792429924011 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9092782139778137 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03580233454704285 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21068359911441803 +particle/sensitivity‚0.6358895301818848 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21045033633708954 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995148241519928 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7685459852218628 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7562949657440186 *jet/accuracy_1_of_2‚0.8101398944854736 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6881068348884583 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6675694584846497 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7681881189346313 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8151730895042419 particle/accuracy_2_of_5‚1 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.728610634803772 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 _step‚263 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8862621784210205 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23497061431407928 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4804818332195282 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04705166071653366 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9081633687019348 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942303895950317 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7735772132873535 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3307197093963623 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.948851466178894 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20797917246818542 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04694057255983353 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8548216223716736 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1Ÿ³ÇÍ ñ{ :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22776247560977936 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0451769083738327 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.740471363067627 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7137414216995239 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44890880584716797 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 /particle/accuracy_3_of_3‚0.9994933605194092 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8739606738090515 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7343984246253967 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.568744957447052 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6268050074577332 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11105038225650787 particle/accuracy_1_of_1‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (jet/accuracy_1_of_4‚0.97369384765625 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29549145698547363 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17836639285087585 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.034710388630628586 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24989673495292664 jet/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6649784445762634 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952138066291809 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3933490514755249 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.554885745048523 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.039802540093660355 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _runtime‚2742.869443121 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6738050580024719 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940407276153564 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7972599864006042 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6255766749382019 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7113804221153259 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 *jet/accuracy_2_of_4‚0.8177792429924011 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9092782139778137 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03580233454704285 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21068359911441803 +particle/sensitivity‚0.6358895301818848 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21045033633708954 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995148241519928 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7685459852218628 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7562949657440186 *jet/accuracy_1_of_2‚0.8101398944854736 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6881068348884583 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6675694584846497 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7681881189346313 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8151730895042419 particle/accuracy_2_of_5‚1 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.728610634803772 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 _step‚263 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8862621784210205 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23497061431407928 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4804818332195282 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04705166071653366 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9081633687019348 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942303895950317 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7735772132873535 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3307197093963623 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.948851466178894 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20797917246818542 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04694057255983353 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8548216223716736 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6675694584846497 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2347719520330429 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14124062657356262 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2012448012828827 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8640019297599792 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929137229919434 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5648747682571411 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04423118755221367 -"Purity/2FRt*SRqqt1FBt/event_purity‚0.1875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5593847036361694 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06653472781181335 *jet/accuracy_3_of_4‚0.0816865786910057 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47300925850868225 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/S1l*Rqqt_purity‚0.7453703284263611 +jet/accuracy_3_of_3‚0.07437843829393387 +jet/accuracy_4_of_4‚0.01628045178949833 'particle/f_score‚0.6453225612640381 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6167340278625488 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23497061431407928 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9938890337944031 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4322916865348816 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12813428044319153 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30262982845306396 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22964322566986084 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_3_of_4‚0.9938492774963379 epoch‚7 (particle/accuracy‚0.8499077558517456 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5512500405311584 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9896675944328308 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6606543064117432 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.039802540093660355 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9965478777885437 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9344983696937561 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6804207563400269 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚8.890356529686532 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872538447380066 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3953187167644501 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4798799157142639 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4597752094268799 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6467170119285583 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *particle/specificity‚0.908441424369812 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7346331477165222 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19811661541461945 /particle/accuracy_4_of_4‚0.9555649757385254 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24517996609210968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19737409055233002 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0830746665596962 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20499177277088165 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9926759600639343 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.899092972278595 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6659560203552246 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18791702389717102 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.043386880308389664 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.786823570728302 /particle/accuracy_2_of_2‚0.9998962879180908 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24697645008563995 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7827073335647583 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953996539115906 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9076341986656189 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04524856060743332 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20878787338733673 *jet/accuracy_2_of_3‚0.5651280879974365 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2954440712928772 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6864126324653625 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.669317901134491 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8548216223716736 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7720421552658081 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1908043920993805 *jet/accuracy_1_of_1‚0.3394206464290619 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7149572968482971 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4471101760864258 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6582632660865784 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9867425560951233 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8811541795730591 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.64238440990448 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_2‚0.1519448310136795 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 % _timestamp‚1.7843602507932901e+09 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24750939011573792 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12395044416189194 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953996539115906 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2187572866678238 6validation_average_jet_accuracy‚0.3818936049938202 trainer/global_step‚3199 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7182226777076721 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7705357670783997 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7621236443519592 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8181013464927673 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961658716201782‡Hý×Êà ‚(:… ¼ÚìÒ”° ¼ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚30.0 gpu.0.powerWatts‚91.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2802 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø~ð=ìÄ ‚(:á ¼ÚìÒ™æý¼( proc.memory.availableMB‚ 742.82421875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 4842.5703125 disk.sda.out‚ 237.18359375 disk.sda2.in‚0" disk.dm-0.out‚10573.8896484375 disk.nvme0n1.in‚ 4842.5703125% disk.nvme0n1.out‚10573.8896484375 disk.sda3.out‚ 237.18359375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 4842.5703125 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625) proc.memory.percent‚16.87145233154297* disk./.usagePercent‚39.525974247935885 disk.dm-1.out‚ 237.18359375 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚ 90.626953125 disk.rbd0.out‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2764.21875 cpu‚26.201606555618508 proc.cpu.threads‚17 network.sent‚ 4.993313e+06 network.recv‚758507$ disk./.usageGB‚588.9655799865723 disk.sda.in‚ 0.00390625 disk.sda2.out‚0' disk.nvme0c0n1.out‚10573.8896484375$ memory_percent‚95.46616077423096q_ã0›Å ‚(: ËÚìÒ¯ÛŸº gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.87519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0póéÆ ‚(:Þ ËÚìÒÔéü½$ memory_percent‚95.46818733215332' proc.memory.availableMB‚ 742.4921875" disk.nvme0c0n1.in‚ 4842.5703125 disk.nvme0n1.in‚ 4842.5703125 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625# disk.dm-0.out‚10642.14697265625 cpu‚26.20086016861017 network.sent‚ 5.050802e+06 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚ 90.626953125 proc.cpu.threads‚17 disk.sda.out‚ 237.3984375 network.recv‚763003& disk.nvme0n1.out‚10642.14697265625 disk.sda1.in‚0 disk.dm-0.in‚ 4842.5703125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.00390625 proc.memory.rssMB‚ 2764.21875 disk.dm-1.out‚ 237.3984375$ disk./.usageGB‚588.9646301269531 disk.rbd0.out‚0 disk.sda2.in‚0) disk./.usagePercent‚39.52591050205986( disk.nvme0c0n1.out‚10642.14697265625 disk.sda1.out‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.87145233154297 disk.sda3.out‚ 237.3984375 disk.sda2.out‚0 disk.drbd2165.in‚0„]W4“Ç ‚(:ˆ ÚÚìÒ¡º gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.132' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à‚Ç.âÈ ‚(:× ÚÚìÒØ‘º¾ disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚10826.3251953125 disk.dm-1.out‚237.6875$ memory_percent‚95.46821117401123 disk.sda3.out‚237.6875 disk.rbd0.out‚0) disk./.usagePercent‚39.52590999004479" disk.dm-0.out‚10826.3251953125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 4896.47265625 disk.sda.out‚237.6875 proc.memory.rssMB‚ 2764.21875! disk.nvme0n1.in‚ 4896.47265625% disk.nvme0n1.out‚10826.3251953125 disk.sda1.out‚0( proc.memory.availableMB‚ 742.48828125 disk.rbd1.out‚ 90.705078125 disk.sda.in‚ 0.00390625 network.sent‚ 5.055814e+06$ disk./.usageGB‚588.9646224975586 disk.dm-1.in‚ 0.00390625 disk.sda2.out‚0 network.recv‚766117 disk.dm-0.in‚ 4896.47265625) proc.memory.percent‚16.87145233154297 cpu‚26.200131581313133 disk.rbd1.in‚ 0.01953125 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0êñx2É - _wandbruntime‚2785  _runtime‚2785Ïø ÉÊ Ã _runtime‚2785.117204304 _step‚264 ' lr-AdamW/pg1‚0.00041271330201480736 ' lr-AdamW/pg2‚0.00041271330201480736 trainer/global_step‚3249 % _timestamp‚1.7843602930418289e+09¥â”LÎË È _runtime‚2785.117204304 _step‚264 ' lr-AdamW/pg1‚0.00041271330201480736 ' lr-AdamW/pg2‚0.00041271330201480736 trainer/global_step‚3249 % _timestamp‚1.7843602930418289e+09ˆvhÓÐ2Ì - _wandbruntime‚2785  _runtime‚2785úDŸ0ß'Í ÙO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367705600 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3438005594112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3437664254976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚491464273113088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25773188 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12690625 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497443726084096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚491464256073728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚3249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664254976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843602930429444e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12690359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494005720489984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55881938 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3438005594112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367705600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494005703450624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112370114270208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108894740891648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3475467984896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚265 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112370208876544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494897033754452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25773190 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25773188 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3432420091474 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5697288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494896676165202 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108894722279424 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚494005703450624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30108748 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3475391990784CКG,'Î ÞO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367705600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494005703450624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112370114270208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108894740891648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3475467984896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚265 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112370208876544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494897033754452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25773190 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25773188 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3432420091474 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5697288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494896676165202 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108894722279424 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚494005703450624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30108748 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3475391990784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367705600 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3438005594112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3437664254976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚491464273113088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25773188 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12690625 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497443726084096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚491464256073728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚3249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664254976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843602930429444e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12690359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494005720489984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55881938 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3438005594112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ p‡¸554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664‰Jš2Ï - _wandbruntime‚2785  _runtime‚2785!˜Ôê&Ð äM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25780820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30117814 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚2785.730520506 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780822 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494152944480256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491610766745600 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30107823 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979747840 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476420930560 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12694089 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5699033 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6995316 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476503208960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676881920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108928169650176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843602936549962e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3438676881920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107823 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚491610783784960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12694352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888643 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3439018228224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108928151037952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621362176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495044555480158 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚3249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5699036 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888643 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3433431145196 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780822 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3433771695198 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780822 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494152944480256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491610766745600 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30107823 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979747840 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476420930560 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12694089 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5699033 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6995316 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476503208960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676881920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108928169650176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843602936549962e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3438676881920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107823 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚491610783784960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12694352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888643 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3439018228224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108928151037952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621362176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495044555480158 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚3249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5699036 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888643 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3433431145196 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780822 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3433771695198 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114689024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154788 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5784538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚504856473067520 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ºøòÒËM=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110566178818048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701095936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12880441 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110566160205824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504856473067520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501367771971584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12880700 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498789789667328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843603337247639e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3483036578382 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708011 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30553223 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3488701095936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3488359756800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26154790 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚498789806706688 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888š®ª¹å'à ßO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26154788 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3488359756800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502273183834960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501367754932224 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30553223 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7095903 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3526843527680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30563200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110566178818048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701095936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12880441 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110566160205824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504856473067520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501367771971584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12880700 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498789789667328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843603337247639e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3483036578382 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708011 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30553223 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3488701095936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3488359756800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26154790 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚498789806706688 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114689024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154788 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5784538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚504856473067520 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2826.415001993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚501515013001216 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182106 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26162422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5786286 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5786283 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚270 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501514995961856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3527870597120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3489713730048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498936300339200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12884205 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26162420 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502420705560666 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110599607576576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734688 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505004726731264 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7098185 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7097922 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30562275 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726731264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3489372383744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114127478173696 trainer/global_step‚3299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498936317378560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110599607576576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734688 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505004726731264 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7098185 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7097922 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30562275 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726731264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3489372383744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114127478173696 trainer/global_step‚3299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498936317378560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2826.415001993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚501515013001216 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182106 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26162422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5786286 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5786283 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚270 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501514995961856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3527870597120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3489713730048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 I@DeviceStatsMonitor.on_tùÂîÖÞrain_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498936300339200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12884205 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26162420 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502420705560666 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861672448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3578159372288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31017652 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396597760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31007675 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7199028 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚508729806413824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13071069 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3533993615180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508729806413824 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57544063 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26536388 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 2866.82002662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3533653065290 % _timestamp‚1.7843603747445016e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506115340300288 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512269220050944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509649333915468 trainer/global_step‚3349 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554042 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112237616744448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509648976326218 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396597760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3539055258624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088‡Ñ=iä'ò ÞO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3539055258624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26536390 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861672448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3578159372288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31017652 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396597760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31007675 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7199028 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚508729806413824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13071069 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3533993615180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508729806413824 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57544063 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26536388 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 2866.82002662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3533653065290 % _timestamp‚1.7843603747445016e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506115340300288 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512269220050944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚509649333915468 trainer/global_step‚3349 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554042 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112237616744448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509648976326218 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396597760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3539055258624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚508729823453184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26536388 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115815776116736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3578064503808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚512268861672448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026718 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚3349 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115850111671808 % _timestamp‚1.7843603753576968e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3534664119012 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540409231872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509796498051812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚512417473714688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3535004669014 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13074538 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚274 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚506261833932800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016727 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26544020 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512417473714688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067885568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31026718 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57570740 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112271045502976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579179642368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579084781056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512417115329024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚508877064482816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269Ì'yî&õ èM F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026718 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚3349 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115850111671808 % _timestamp‚1.7843603753576968e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3534664119012 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540409231872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509796498051812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚512417473714688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3535004669014 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13074538 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚274 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚506261833932800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016727 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26544020 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512417473714688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067885568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31026718 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57570740 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112271045502976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579179642368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579084781056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512417115329024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚508877064482816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560747 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5873533 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7201286 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976’*Vx2ö - _wandbruntime‚2867  _runtime‚2867ýÜ]÷ ü 1loss/SRqqt2/assignment_loss‚0.591255928857936 /loss/FBt1/assignment_loss‚0.207884582615367 $loss/total_loss‚9.14389591111209 0loss/FRt2/assignment_loss‚0.7728872366374631 /loss/FRt1/detection_loss‚0.5337996697966735 /loss/FRt2/detection_loss‚1.4799391869321572 1loss/SRqqt2/detection_loss‚0.6488514096805981 .loss/FBt2/detection_loss‚0.523720601502658 _runtime‚2867.435191087 /loss/FRt1/assignment_loss‚1.882932555043336 2loss/SRqqt1/assignment_loss‚0.5702838743755737 2loss/FBt2/assignment_loss‚0.023869273249433373 1loss/SRqqt1/detection_loss‚0.9942767097759847 _step‚275 /loss/FBt1/detection_loss‚0.9141948826449073 epoch‚8 trainer/global_step‚3349 % _timestamp‚1.7843603753601089e+092›z*ø  1loss/SRqqt1/detection_loss‚0.9942767097759847 _step‚275 /loss/FBt1/detection_loss‚0.9141948826449073 epoch‚8 trainer/global_step‚3349 % _timestamp‚1.7843603753601089e+09 1loss/SRqqt2/assignment_loss‚0.591255928857936 /loss/FBt1/assignment_loss‚0.207884582615367 $loss/total_loss‚9.14389591111209 0loss/FRt2/assignment_loss‚0.7728872366374631 /loss/FRt1/detection_loss‚0.5337996697966735 /loss/FRt2/detection_loss‚1.4799391869321572 1loss/SRqqt2/detection_loss‚0.6488514096805981 .loss/FBt2/detection_loss‚0.523720601502658 _runtime‚2867.435191087 /loss/FRt1/assignment_loss‚1.882932555043336 2loss/SRqqt1/assignment_loss‚0.5702838743755737 2loss/FBt2/assignment_loss‚0.023869273249433373“§ëÈéù ‚(:„ ÃÛìÒܬù» gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö‚‚³éú ‚(:Þ ÃÛìÒ£Œ½ disk.drbd2165.in‚0 disk.sda1.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚10858.0068359375 disk.nvme0n1.in‚ 4946.796875 disk.sda.out‚ 240.05078125 network.recv‚791851 proc.memory.rssMB‚ 2764.21875 disk.sda.in‚ 0.00390625$ disk./.usageGB‚588.9648284912109 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01953125 disk.sda3.out‚ 240.05078125" disk.dm-0.out‚10858.0068359375 disk.dm-1.out‚ 240.05078125% disk.nvme0n1.out‚10858.0068359375 proc.cpu.threads‚17 disk.drbd2165.out‚0 network.sent‚ 5.205561e+06) disk./.usagePercent‚39.52592381445164 disk.dm-0.in‚ 4946.796875 disk.rbd0.out‚0( proc.memory.availableMB‚ 495.33203125$ memory_percent‚96.97673320770264 disk.sda3.in‚ 0.00390625 disk.rbd1.out‚90.93212890625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 4946.796875) proc.memory.percent‚16.87145233154297 disk.dm-1.in‚ 0.00390625 cpu‚26.19231943059527ác#û ‚(:’ ÒÛìÒŸò÷¹ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.184' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.63680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËA…ëü ‚(:à ÒÛìÒ‹²¾ disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 proc.memory.rssMB‚ 2764.21875" disk.nvme0c0n1.in‚ 5017.3828125 disk.drbd2165.in‚0' proc.memory.availableMB‚ 495.5546875 network.sent‚ 5.26829e+06# disk.dm-0.out‚10861.04443359375 cpu‚26.191567437648096 network.recv‚795938& disk.nvme0n1.out‚10861.04443359375 disk.sda3.out‚ 240.3828125) proc.memory.percent‚16.87145233154297) disk./.usagePercent‚39.52593431076054 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚91.07275390625 disk.sda.out‚ 240.3828125 disk.drbd2165.out‚0$ memory_percent‚96.97537422180176 proc.cpu.threads‚17$ disk./.usageGB‚588.9649848937988 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10861.04443359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 240.3828125 disk.sda.in‚ 0.00390625 disk.sda1.out‚0 disk.dm-0.in‚ 5017.3828125 disk.rbd1.in‚ 0.01953125 disk.nvme0n1.in‚ 5017.3828125÷àý%2ý - _wandbruntime‚2907  _runtime‚2907f¸y*Çþ Á _step‚276 & lr-AdamW/pg1‚0.0004103512604245744 & lr-AdamW/pg2‚0.0004103512604245744 trainer/global_step‚3399 % _timestamp‚1.7843604154527798e+09 _runtime‚2907.531232321‘…ôûÌÿ Æ & lr-AdamW/pg2‚0.0004103512604245744 trainer/global_step‚3399 % _timestamp‚1.7843604154527798e+09 _runtime‚2907.531232321 _step‚276 & lr-AdamW/pg1‚0.0004103512604245744”à8¾`2€- _wandbruntime‚2907  _runtime‚29075ëìà'ÚO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3590092099584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26917990 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26917990 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31472104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113909054670848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513440856854528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13261800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5959038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390094 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608655872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31462127 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7302759 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚2907.532241537 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3589750760448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7302499 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117538529261568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608655872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31462127 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7302759 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 56413388ã­Rð3 8 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚2907.532241537 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3589750760448 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7302499 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117538529261568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3590092099584 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26917990 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26917990 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31472104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113909054670848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513440856854528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13261800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5959038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390094 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7304778 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517172648132320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚2908.334574355 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960786 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843604162589996e+09 _step‚278 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13265294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26925622 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3590763387392 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3630494057984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚513587384565760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3630399196672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3585280605920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117572977487360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3590763387392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519830220698112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26925620 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31471179 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3591104733696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31481170 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7304511 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26925620 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31481170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513587367526400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155922 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516239098925056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104733696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31471179 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396799 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517173005721682 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7304778 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517172648132320 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚2908.334574355 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960786 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843604162589996e+09 _step‚278 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13265294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26925622 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3590763387392 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3630494057984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚513587384565760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3630399196672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3585280605920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117572977487360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3590763387392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519830220698112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26925620 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31471179 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3591104733696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31481170 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7304511 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26925620 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31481170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513587367526400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155922 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516239098925056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104733696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31471179 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396799 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517173005721682 EH‡ 0loss/FRt2/assignment_loss‚0.6852522228283905 /loss/FRt2/detection_loss‚1.5713915140963914 $loss/total_loss‚8.85206847072865 1loss/SRqqt1/detection_loss‚0.9528595195935652 _runtime‚2908.335813975 2loss/SRqqt1/assignment_loss‚0.7041852968108971 2loss/FBt1/assignment_loss‚0.049928725387744594 /loss/FRt1/detection_loss‚0.5763766456920288 1loss/SRqqt2/detection_loss‚0.6392152341465461 /loss/FBt1/detection_loss‚0.6298876029128829 trainer/global_step‚3399 % _timestamp‚1.7843604162607281e+09 0loss/FRt1/assignment_loss‚1.8835432945349706 2loss/SRqqt2/assignment_loss‚0.5908033221358315 2loss/FBt2/assignment_loss‚0.023272580591349293 .loss/FBt2/detection_loss‚0.545352511998051 epoch‚8 _step‚279´E ˆ† /loss/FRt2/detection_loss‚1.5713915140963914 $loss/total_loss‚8.85206847072865 1loss/SRqqt1/detection_loss‚0.9528595195935652 _runtime‚2908.335813975 2loss/SRqqt1/assignment_loss‚0.7041852968108971 2loss/FBt1/assignment_loss‚0.049928725387744594 /loss/FRt1/detection_loss‚0.5763766456920288 1loss/SRqqt2/detection_loss‚0.6392152341465461 /loss/FBt1/detection_loss‚0.6298876029128829 trainer/global_step‚3399 % _timestamp‚1.7843604162607281e+09 0loss/FRt1/assignment_loss‚1.8835432945349706 2loss/SRqqt2/assignment_loss‚0.5908033221358315 2loss/FBt2/assignment_loss‚0.023272580591349293 .loss/FBt2/detection_loss‚0.545352511998051 epoch‚8 _step‚279 0loss/FRt2/assignment_loss‚0.6852522228283905—‡¶ž‰‚(:“ áÛìÒãÌ» gpu.0.gpu‚88.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘êpAàŠ‚(:Õ áÛìÒýñ¶¼( disk.nvme0c0n1.out‚10865.24560546875& disk.nvme0n1.out‚10865.24560546875 disk.drbd2165.out‚0 disk.dm-0.in‚ 5017.3828125) proc.memory.percent‚16.87145233154297 network.sent‚ 5.2733e+06 disk.rbd1.out‚91.07275390625 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 cpu‚26.190535742239394# disk.dm-0.out‚10865.24560546875 disk.sda.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 disk.sda1.in‚0 disk.nvme0n1.in‚ 5017.3828125 disk.sda3.out‚240.875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚96.96314334869385 disk.drbd2165.in‚0( proc.memory.availableMB‚ 497.55859375 proc.memory.rssMB‚ 2764.21875 proc.cpu.threads‚17 disk.dm-1.out‚240.875 disk.sda.out‚240.875 network.recv‚799203* disk./.usagePercent‚39.525931494677664$ disk./.usageGB‚588.9649429321289" disk.nvme0c0n1.in‚ 5017.3828125iåh’‹‚(:‡ ðÛìÒñÇõº gpu.0.gpu‚23.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘÷Ý쌂(:á ðÛìÒ€åâ½ disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚91.13525390625$ memory_percent‚96.96037769317627( proc.memory.availableMB‚ 498.01171875( disk.nvme0c0n1.out‚10868.34814453125 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5017.3828125 proc.memory.rssMB‚ 2764.21875# disk./.usageGB‚588.964900970459 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625 cpu‚26.189796244826315 network.sent‚ 5.336086e+06 disk.dm-1.out‚ 241.4453125) proc.memory.percent‚16.87145233154297 proc.cpu.threads‚17) disk./.usagePercent‚39.52592867859479 disk.nvme0n1.in‚ 5017.3828125& disk.nvme0n1.out‚10868.34814453125 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 network.recv‚803363" disk.nvme0c0n1.in‚ 5017.3828125 disk.sda.out‚ 241.4453125 disk.sda3.out‚ 241.4453125 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚10868.34814453125 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0•LûÍ-‚(:’ ÿÛìÒíªç» gpu.0.gpu‚97.03³ˆFp gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚268.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.767999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z¿Ù7厂(:Ú ÿÛìÒ´Û‚½* disk./.usagePercent‚39.525848804244106 disk.sda3.out‚ 241.77734375 proc.memory.rssMB‚ 2764.21875# disk.nvme0n1.out‚10875.07421875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 disk.rbd0.out‚0 network.recv‚806615 disk.dm-1.in‚ 0.00390625% disk.nvme0c0n1.out‚10875.07421875 disk.sda.in‚ 0.00390625 disk.dm-1.out‚ 241.77734375 disk.rbd0.in‚0) proc.memory.percent‚16.87145233154297 cpu‚26.18859674890393 disk.sda1.in‚0 disk.dm-0.in‚ 5017.3828125 disk.rbd1.in‚ 0.01953125 disk.nvme0n1.in‚ 5017.3828125 disk.dm-0.out‚10875.07421875$ memory_percent‚96.96264266967773 network.sent‚ 5.341158e+06$ disk./.usageGB‚588.9637107849121 disk.sda.out‚ 241.77734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚91.13525390625" disk.nvme0c0n1.in‚ 5017.3828125& proc.memory.availableMB‚ 497.640625 8†A2- _wandbruntime‚2948  _runtime‚2948Î åÎÇÁ _step‚280 & lr-AdamW/pg1‚0.0004095390720345193 & lr-AdamW/pg2‚0.0004095390720345193 trainer/global_step‚3449 % _timestamp‚1.7843604563377867e+09 _runtime‚2948.413159922Æ5xÌ‘Æ trainer/global_step‚3449 % _timestamp‚1.7843604563377867e+09 _runtime‚2948.413159922 _step‚280 & lr-AdamW/pg1‚0.0004095390720345193 & lr-AdamW/pg2‚0.0004095390720345193˜õu§2’- _wandbruntime‚2948  _runtime‚2948øU1á'“ÛO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787601408 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520766407487488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27299588 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3640446262272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115580492597248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115580473985024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216167 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523453926416384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31916579 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3640787601408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916579 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3680674151424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119261046976512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27299590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7405339 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚281 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚527094355639296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13451916 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6046288 % _timestamp‚1.7843604563386102e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31926556 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3635226588996 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523453909377024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523453926416384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚527094714017792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119261166748672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3680572991488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524401634076484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3634886039106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355639296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27299590 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226146 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888u¿±Åæ'”àO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚2948.414083152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31926556 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31916579 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3640787601408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916579 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3680674151424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119261046976512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27299590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7405339 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚281 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚527094355639296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13451916 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6046288 % _timestamp‚1.7843604563386102e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31926556 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3635226588996 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523453909377024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523453926416384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚527094714017792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119261166748672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3680572991488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524401634076484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3634886039106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355639296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27299590 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226146 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787601408 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520766407487488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27299588 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3640446262272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115580492597248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115580473985024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216167 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523453926416384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3641458889216 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13455377 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚520912918159360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚527242967681536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31925631 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31935622 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523601167446016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27307222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119295612498432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3681691142656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3636237642830 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232851 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚3449 _step‚282 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3641800235520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115613902743552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3681604669952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843604569546015e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6048033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523601167446016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27307220 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3635897092828 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27307222 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚527242609295872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119295507413504 _runtime‚2949.030246335 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7407344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524548798212828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520912901120000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115613921355776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925631 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13455656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3641458889216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7407620 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3641458889216 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13455377 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚520912918159360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚527242967681536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31925631 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31935622 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523601167446016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27307222 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119295612498432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3681691142656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3636237642830 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232851 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚3449 _step‚282 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3641800235520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115613902743552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3681604669952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7843604569546015e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6048033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523601167446016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27307220 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3635897092828 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27307222 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚527242609295872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119295507413504 _runtime‚2949.030246335 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7407344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524548798212828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520912901120000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115613921355776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925631 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13455656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3641458889216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7407620 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0š\.3Ñ2˜- _wandbruntime‚2949  _runtime‚29495!•™ 1loss/FBt2/assignment_loss‚0.03467796441657651 /loss/FRt1/detection_loss‚0.5106623296742294 /loss/FRt2/detection_loss‚1.5389244529131916 1loss/SRqqt2/detection_loss‚0.6750472579814708 /loss/FBt1/detection_loss‚0.9959053790437118 trainer/global_step‚3449 _step‚283 /loss/FRt2/assignment_loss‚0.751191143204318 2loss/SRqqt1/assignment_loss‚0.6510461771841182 2loss/SRqqt2/assignment_loss‚0.5254211854256897 epoch‚8 0loss/FRt1/assignment_loss‚1.9068751729401077 2loss/FBt1/assignment_loss‚0.043077637587327536 1loss/SRqqt1/detection_loss‚0.9824091142501076 /loss/FBt2/detection_loss‚0.5925170556642115 %loss/total_loss‚9.207754870285061 % _timestamp‚1.7843604569569316e+09 _runtime‚2949.032010691`ø»I š† %loss/total_loss‚9.207754870285061 % _timestamp‚1.7843604569569316e+09 _runtime‚2949.032010691 1loss/FBt2/assignment_loss‚0.03467796441657651 /loss/FRt1/detection_loss‚0.5106623296742294 /loss/FRt2/detection_loss‚1.5389244529131916 1loss/SRqqt2/detection_loss‚0.6750472579814708 /loss/FBt1/detection_loss‚0.9959053790437118 trainer/global_step‚3449 _step‚283 /loss/FRt2/assignment_loss‚0.751191143204318 2loss/SRqqt1/assignment_loss‚0.6510461771841182 2loss/SRqqt2/assignment_loss‚0.5254211854256897 epoch‚8 0loss/FRt1/assignment_loss‚1.9068751729401077 2loss/FBt1/assignment_loss‚0.043077637587327536 1loss/SRqqt1/detection_loss‚0.9824091142501076 /loss/FBt2/detection_loss‚0.5925170556642115›‡Z"»“›‚(:ˆ ŽÜìÒú·º gpu.0.gpu‚47.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–n*㜂(:Ø ŽÜìÒ¹š„½$ disk.nvme0n1.out‚10879.587890625 disk.sda.out‚ 242.05859375' proc.memory.availableMB‚ 497.4140625 proc.cpu.threads‚17 disk.rbd1.out‚91.21337890625 disk.drbd2165.in‚0 network.sent‚ 5.346169e+06& disk.nvme0c0n1.out‚10879.587890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 5081.28125 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 242.05859375 cpu‚26.187814720784054 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 242.05859375 disk.drbd2165.out‚0$ memory_percent‚96.96402549743652 proc.memory.rssMB‚ 2764.21875 disk.sda.in‚ 0.00390625 network.recv‚809577 disk.rbd1.in‚ 0.01953125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0$ disk./.usageGB‚588.9647026062012 disk.rbd0.in‚0! disk.dm-0.out‚10879.587890625) proc.memory.percent‚16.87145233154297) disk./.usagePercent‚39.52591536620301 disk.nvme0c0n1.in‚ 5081.28125 disk.nvme0n1.in‚ 5081.28125¿Â×½ž‚(:“ ÜìÒ¹Ñʺ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.717200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04¢Q«åž‚(:Ú ÜìÒÉÊȼ disk.sda.out‚ 242.2265625 disk.sda2.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚10884.8974609375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.01953125( proc.memory.availableMB‚ 497.47265625 network.sent‚ 5.409321e+06 network.recv‚814469% disk.nvme0n1.out‚10884.8974609375 disk.drbd2165.out‚0 disk.dm-0.in‚ 5081.28125* disk./.usagePercent‚39.525912806127664) proc.memory.percent‚16.87145233154297 disk.nvme0n1.in‚ 5081.28125 disk.sda3.out‚ 242.2265625 disk.sda1.in‚0$ disk./.usageGB‚588.9646644592285 disk.sda2.in‚0' disk.nvme0c0n1.out‚10884.8974609375$ memory_percent‚96.96366786956787 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 242.2265625 cpu‚26.187967690125532 disk.sda3.in‚ 0.00390625 disk.rbd1.out‚91.21337890625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 5081.28125 proc.memory.rssMB‚ 2764.218755‰(“Ÿ‚(:ˆ ¬ÜìÒ«¯æ¼ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚253.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™IB8ã ‚(:Ø ¬ÜìÒ´’í½! disk.dm-0.out‚10891.888671875 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚91.27587890625 disk.sda3.out‚ 243.47265625# disk./.usageGB‚588.965274810791 disk.nvme0c0n1.in‚ 5081.28125$ disk.nvme0n1.out‚10891.888671875 disk.sda.in‚ 0.00390625 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.52595376733314 disk.rbd0.in‚0 disk.nvme0n1.in‚ 5081.28125 disk.sda2.in‚0$ memory_percent‚96.96099758148193 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚10891.888671875) proc.memory.percent‚16.87145233154297 disk.sda1.out‚0( proc.memory.availableMB‚ 497.91015625 disk.rbd0.out‚0 network.sent‚ 5.414326e+06 disk.dm-0.in‚ 5081.28125 disk.dm-1.in‚ 0.00390625 network.recv‚817754 disk.dm-1.out‚ 243.47265625 disk.sda.out‚ 243.47265625 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 cpu‚26.186758424992263 proc.memory.rssMB‚ 2764.21875VLƒ2¡- _wandbruntime‚2989  _runtime‚2989ŒÂWɢà % _timestamp‚1.7843604972577872e+09 _runtime‚2989.335221901 _step‚284 ' lr-AdamW/pg1‚0.00040871454932652927 ' lr-AdamW/pg2‚0.00040871454932652927 trainer/global_step‚3499¶\š±Î£È % _timestamp‚1.7843604972577872e+09 _runtime‚2989.335221901 _step‚284 ' lr-AdamW/pg1‚0.00040871454932652927 ' lr-AdamW/pg2‚0.00040871454932652927 trainer/global_step‚3499œÊÕ(É2¤- _wandbruntime‚2989  _runtime‚2989«£ß'¥ÙO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534507461001216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3691141764096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13641526 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102622720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3691483103232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3691141764096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32371031 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3685843075904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6133541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6133538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7508250 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7507988 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117251911911424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32381008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32381008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528091924041728 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3731784963072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3731875637248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27681190 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117251930523648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531777426567742 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13641791 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531777784156992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888Ñ((—¦ÞO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534507461001216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3691141764096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13641526 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102622720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3691483103232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3691141764096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32371031 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3685843075904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6133541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6133538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7508250 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7507988 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117251911911424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32381008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32381008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚528091924041728 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3731784963072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3731875637248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27681190 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117251930523648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531777426567742 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13641791 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531777784156992¾¢±ƒ2§- _wandbruntime‚2989  _runtime‚2989È86Né&¨ãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27688820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117285359282176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068903 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534655714664960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530963218927616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530963201888256 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32390074 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534655714664960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068903 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3692154391040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692154391040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3692495737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13645308 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531924948293336 % _timestamp‚1.7843604978773887e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚530963218927616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3732894858752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚530963201888256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13645573 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2989.95301474 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534655356279296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531925305882698 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27688822 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 _step‚286 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3692495737344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27688822 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32380083 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7510287 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚3499 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7510025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528238434713600 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530963218927616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530963201888256 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32390074 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534655714664960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068903 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3692154391040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692154391040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3692495737344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13645308 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531924948293336 % _timestamp‚1.7843604978773887e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚530963218927616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3732894858752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚530963201888256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13645573 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2989.95301474 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534655356279296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531925305882698 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27688822 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 _step‚286 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3692495737344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27688822 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32380083 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7510287 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚3499 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7510025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528238434713600 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438ž,…/讂(:Ý »ÜìÒ‹åà½# disk.dm-0.out‚10898.86669921875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 5081.28125( disk.nvme0c0n1.out‚10898.86669921875( proc.memory.availableMB‚ 497.74609375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.dm-1.out‚ 244.8203125 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 5.419194e+06 network.recv‚821451$ disk./.usageGB‚588.9662017822266 disk.rbd1.out‚91.27587890625 disk.sda3.out‚ 244.8203125) proc.memory.percent‚16.87145233154297* disk./.usagePercent‚39.526015977163965 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 244.8203125 disk.sda2.in‚0 disk.dm-0.in‚ 5081.28125 disk.nvme0n1.in‚ 5081.28125& disk.nvme0n1.out‚10898.86669921875$ memory_percent‚96.96199893951416 proc.memory.rssMB‚ 2764.21875 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0 cpu‚26.186081644961444ÑÀ®¯‚(:’ ÊÜìÒõûä» gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m>wÜ°‚(:Ñ ÊÜìÒÙù•½) disk./.usagePercent‚39.52600957697561 disk.rbd1.in‚ 0.01953125# disk.nvme0n1.out‚10901.66015625 proc.memory.rssMB‚ 2764.21875 disk.dm-1.out‚ 245.421875$ disk./.usageGB‚588.9661064147949 disk.drbd2165.in‚0 disk.rbd1.out‚91.35400390625" disk.nvme0c0n1.in‚ 5156.6796875 disk.sda.out‚ 245.421875 disk.sda3.in‚ 0.00390625) proc.memory.percent‚16.87145233154297 cpu‚26.18550289558757 disk.dm-0.in‚ 5156.6796875 disk.dm-0.out‚10901.66015625 disk.dm-1.in‚ 0.00390625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0$ proc.memory.availableMB‚497.9375 network.sent‚ 5.482207e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 245.421875$ memory_percent‚96.96083068847656 network.recv‚825694 disk.sda1.out‚0% disk.nvme0c0n1.out‚10901.66015625 disk.nvme0n1.in‚ 5156.6796875 proc.cpu.threads‚17jK”ž±‚(:“ ÙÜìÒʤһ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=KÆ0ﲂ(:ä ÙÜìÒ€Ó¼" disk.nvme0c0n1.in‚ 5156.6796875$ memory_percent‚96.96135520935059 cpu‚26.184764888842746 disk.rbd0.in‚0 disk.sda.out‚ 245.75390625# disk.dm-0.out‚10907.80419921875 disk.dm-1.in‚ 0.00390625) proc.memory.percent‚16.87145233154297 disk.dm-1.out‚ 245.75390625 proc.cpu.threads‚17$ disk./.usageGB‚588.9662475585938 disk.rbd1.in‚ 0.01953125' proc.memory.availableMB‚ 497.8515625 disk.rbd1.out‚91.35400390625 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚10907.80419921875 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2764.21875& disk.nvme0n1.out‚10907.80419921875 disk.sda.in‚ 0.00390625 disk.sda1.in‚0 disk.drbd2165.in‚0 network.sent‚ 5.487078e+06 disk.nvme0n1.in‚ 5156.6796875 disk.sda3.out‚ 245.75390625 disk.sda2.out‚0 disk.dm-0.in‚ 5156.6796875 network.recv‚829381) disk./.usagePercent‚39.52601904925437 disk.rbd0.out‚0¼;ðõ2³- _wandbruntime‚3030  _runtime‚3030˜=ð É´Ã ' lr-AdamW/pg2‚0.00040787774525815723 trainer/global_step‚3549 % _timestamp‚1.7843605380158205e+09 _runtime‚3030.094235491 _step‚288 ' lr-AdamW/pg1‚0.00040787774525815723œ ícÎµÈ _step‚288 ' lr-AdamW/pg1‚0.00040787774525815723 ' lr-AdamW/pg2‚0.00040787774525815723 trainer/global_step‚3549 % _timestamp‚1.7843605380158205e+09 _runtime‚3030.094235491  ¢[*2¶- _wandbruntime‚3030  _runtime‚3030 ËÈ[` ·ÚO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7610847 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849606144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32825483 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocaÃël€ted‚ 3783080492032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541920207984640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3742178605056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782989817856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13831638 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32835460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28062790 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚535417474674688 % _timestamp‚1.7843605380197392e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32835460 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13831375 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28062790 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118923349837824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122706448942080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538178029379584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118923368450048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28062788 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207984640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3741837265920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976L§§Yå'¸ßO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539153934237500 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849606144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32825483 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3783080492032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541920207984640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3742178605056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782989817856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13831638 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32835460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28062790 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚535417474674688 % _timestamp‚1.7843605380197392e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32835460 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13831375 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28062790 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118923349837824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122706448942080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538178029379584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118923368450048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28062788 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207984640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3741837265920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7610847 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32844526 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32844526 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122740899912192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28070422 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538325270409216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32834535 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963206 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚535563968307200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3742849892864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28070420 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539301098373844 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3743191239168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542068461648384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3784102703616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚535563985346560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7612928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118956797208576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 $ _timestamp‚1.784360538632325e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6222533 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3030.707898209 _step‚290 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60904955 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3737470616646 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3742849892864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28070420 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538325253369856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7612663 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3737130066644 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3784014133760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542068103262720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13835464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122740792730112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13835196 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚3549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6222536›c"¹]»èM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3743191239168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚542068461648384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3784102703616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚535563985346560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7612928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118956797208576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 $ _timestamp‚1.784360538632325e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6222533 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3030.707898209 _step‚290 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60904955 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3737470616646 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3742849892864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28070420 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538325253369856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7612663 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3737130066644 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3784014133760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542068103262720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13835464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122740792730112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13835196 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚3549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6222536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32844526 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32844526 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122740899912192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28070422 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538325270409216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32834535 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963206 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚535563968307200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3742849892864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28070420 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539301098373844¢Y—}N2¼- _wandbruntime‚3030  _runtime‚3030ûÌÙ*½€ /loss/FBt1/detection_loss‚1.0069024087540035 0loss/FBt2/detection_loss‚0.48467443145575445 $loss/total_loss‚9.03697428999741 epoch‚8 trainer/global_step‚3549 _runtime‚3030.710203652 1loss/SRqqt2/detection_loss‚0.6352322244415215 0loss/FRt2/assignment_loss‚0.6354121747471038 2loss/SRqqt1/assignment_loss‚0.6475257705281818 2loss/SRqqt2/assignment_loss‚0.5241035257516362 1loss/FBt1/assignment_loss‚0.17775728914198383 /loss/FRt1/detection_loss‚0.5500838379999444 /loss/FRt2/detection_loss‚1.4532720930186942 1loss/SRqqt1/detection_loss‚1.0139760226188335 $ _timestamp‚1.784360538635158e+09 _step‚291 /loss/FRt1/assignment_loss‚1.883233335776452 2loss/FBt2/assignment_loss‚0.024801175763301576>ʱ¼ ¾… /loss/FRt1/assignment_loss‚1.883233335776452 2loss/FBt2/assignment_loss‚0.024801175763301576 /loss/FBt1/detection_loss‚1.0069024087540035 0loss/FBt2/detection_loss‚0.48467443145575445 $loss/total_loss‚9.03697428999741 epoch‚8 trainer/global_step‚3549 _runtime‚3030.710203652 1loss/SRqqt2/detection_loss‚0.6352322244415215 0loss/FRt2/assignment_loss‚0.6354121747471038 2loss/SRqqt1/assignment_loss‚0.6475257705281818 2loss/SRqqt2/assignment_loss‚0.5241035257516362 1loss/FBt1/assignment_loss‚0.17775728914198383 /loss/FRt1/detection_loss‚0.5500838379999444 /loss/FRt2/detection_loss‚1.4532720930186942 1loss/SRqqt1/detection_loss‚1.0139760226188335 $ _timestamp‚1.784360538635158e+09 _step‚291£cÛÌ\ë¿‚(:à èÜìÒ›‰ú¼ disk.sda1.in‚0" disk.nvme0c0n1.in‚ 5156.6796875 disk.sda3.out‚ 246.3828125 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0& disk.nvme0n1.out‚10911.13134765625 network.sent‚ 5.492088e+06 disk.dm-1.out‚ 246.3828125( disk.nvme0c0n1.out‚10911.13134765625 disk.nvme0n1.in‚ 5156.6796875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚39.52601802522423 disk.sda2.out‚0# disk.dm-0.out‚10911.13134765625' proc.memory.availableMB‚ 497.2890625 cpu‚26.18434533691776 disk.sda.out‚ 246.3828125$ memory_percent‚96.96478843688965 proc.memory.rssMB‚ 2764.21875) proc.memory.percent‚16.87145233154297 network.recv‚832741 disk.dm-0.in‚ 5156.6796875 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625$ disk./.usageGB‚588.9662322998047 disk.rbd1.out‚91.47900390625 disk.sda.in‚ 0.00390625 proc.cpu.threads‚17Ñg}¥žÀ‚(:“ èÜìÒß®À½ gpu.0.gpu‚41.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07–2&‘Á‚(:† ÷ÜìÒâþŠ» gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³T5å‚(:Ú ÷ÜìÒÓ’¬¾ disk.rbd0.out‚0' disk.nvme0c0n1.out‚10915.6962890625 cpu‚26.18366719302322 proc.cpu.threads‚17 network.sent‚ 5.554841e+06 disk.sda.out‚ 246.828125 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 246.828125 disk.sda1.out‚0" disk.dm-0.out‚10915.6962890625 disk.rbd1.out‚91.47900390625 disk.dm-0.in‚ 5156.6796875 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.01953125) disk./.usagePercent‚39.52600778492287 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚96.96440696716309) proc.memory.percent‚16.87145233154297% disk.nvme0n1.out‚10915.6962890625 disk.drbd2165.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 5156.6796875 proc.memory.rssMB‚ 2764.21875 network.recv‚837180 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625' proc.memory.availableMB‚ 497.3515625$ disk./.usageGB‚588.9660797119141 disk.nvme0n1.in‚ 5156.6796875 disk.dm-1.out‚ 246.828125òã¸þ2Ã- _wandbruntime‚3070  _runtime‚3070œ?%}ÇÄÁ % _timestamp‚1.7843605789088678e+09 _runtime‚3070.984164089 _step‚292 & lr-AdamW/pg1‚0.0004070287135757647 & lr-AdamW/pg2‚0.0004070287135757647 trainer/global_step‚3599ZüÒÌÅÆ trainer/global_step‚3599 % _timestamp‚1.7843605789088678e+09 _runtime‚3070.984164089 _step‚292 & lr-AdamW/pg1‚0.0004070287135757647 & lr-AdamW/pg2‚0.0004070287135757647¤nz!2Æ- _wandbruntime‚3070  _runtime‚3070"Q¼Zß'ÇÙO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120594806376448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724323 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28444388 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14021412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596589568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚ 3070.98553661 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33289912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28444390 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33289912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3792532767744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532767744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3787076049720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3786735499830 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549332596589568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6308041 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚546530084318008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542742991228928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3792874106880 % _timestamp‚1.7843605789099555e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545540063821824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545540063821824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6308038 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3599 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542743008268288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549332954968064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874106880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7713374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728758 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124428997522944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚293 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7713639 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14021680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124429100517888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120594787764224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3834294141440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549332954968064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33279935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596589568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚ 3070.98553661 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33289912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28444390 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33289912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3792532767744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532767744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3787076049720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3786735499830 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549332596589568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6308041 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚546530084318008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542742991228928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3792874106880 % _timestamp‚1.7843605789099555e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545540063821824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545540063821824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6308038 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3599 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542743008268288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549332954968064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874106880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7713374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728758 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124428997522944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚293 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7713639 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14021680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734302 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124429100517888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120594787764224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3834294141440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549332954968064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33279935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545687321890816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14025388 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatÄ¢ âsMonitor.on_train_batch_end/active.all.freed‚61741007 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3793886740992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3793545394688 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7715350 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120628216522752 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3788087103554 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61751000 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124463449425408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚294 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6309786 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124463552413184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545687304851456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7715602 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33298978 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28452020 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33298978 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14025133 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚545687304851456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741007 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3793545394688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3835232902656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28452022 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3835317278208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542889518940160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3787746553552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546677606043714 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E‚ìZî&ËèM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3788087103554 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61751000 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124463449425408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚294 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6309786 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124463552413184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545687304851456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7715602 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33298978 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28452020 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33298978 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14025133 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚545687304851456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741007 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3793545394688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3835232902656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28452022 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3835317278208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542889518940160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3787746553552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546677606043714 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545687321890816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14025388 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741007 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3793886740992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3793545394688 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7715350 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120628216522752 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 P&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_1‚0.3471159338951111 (particle/accuracy‚0.8526204228401184 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 % _timestamp‚1.7843605980898123e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7594316601753235 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6110759377479553 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2019769847393036 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9929133653640747 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6195901036262512 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18583333492279053 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8473358154296875 particle/accuracy_1_of_4‚1 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.98603355884552 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3122864365577698 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7323473691940308 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953984022140503 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24564072489738464 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953984022140503 +jet/accuracy_2_of_2‚0.15570756793022156 +jet/accuracy_3_of_4‚0.09999000281095505 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5572516322135925 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26383528113365173 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24542947113513947 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4849550127983093 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3599 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7430240511894226 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7102619409561157 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6763437986373901 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6812182068824768 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19197121262550354 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9925982356071472 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2640424966812134 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7583575248718262 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04075905308127403 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9894558191299438 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22943246364593506 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859439134597778 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13742682337760925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20802223682403564 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8473358154296875 +jet/accuracy_3_of_3‚0.07372218370437622 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20702488720417023 _step‚296 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5520271062850952 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955244660377502 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9265109300613403 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12201228737831116 *jet/accuracy_2_of_4‚0.8141986131668091 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936091899871826 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6208056211471558 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0916481539607048 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.666715681552887 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6623740196228027 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2249438613653183 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 epoch‚8 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03939174860715866 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6743825078010559 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0423104465007782 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860796332359314 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4554497003555298 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22502022981643677 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.895575761795044 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7549886703491211 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7665966749191284 *jet/accuracy_1_of_4‚0.9682220220565796 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2642631530761719 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4686416983604431 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8768740892410278 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3957784175872803 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995604395866394 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7618318796157837 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.550156831741333 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4427083432674408 _runtime‚3090.165811739 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6697901487350464 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6338476538658142 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.835355281829834 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07586878538131714 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04107978567481041 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8607978820800781 ,jet/accuracy_4_of_4‚0.013899499550461769 6validation_average_jet_accuracy‚0.3877866864204407 particle/accuracy_3_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24835191667079926 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2530174255371094 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9327751398086548 &ê‚»O Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047986030578613 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6822916865348816 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04982021823525429 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7630487084388733 *jet/accuracy_2_of_3‚0.5696191787719727 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7618455290794373 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8280045390129089 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6675854921340942 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6152611970901489 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8450114130973816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6892662048339844 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7076148986816406 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_3_of_4‚1 particle/accuracy_4_of_5‚1 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992159366607666 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7151239514350891 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39753296971321106 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6484599709510803 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9103319048881531 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.650034487247467 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962165355682373 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44615691900253296 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8017535209655762 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2652738690376282 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.996662437915802 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7270690202713013 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7634421586990356 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047763172537088394 .particle/accuracy_4_of_4‚0.954102098941803 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995124101638794 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2530174255371094 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039602428674697876 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0423104465007782 particle/accuracy_2_of_3‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47048649191856384 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8223734498023987 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5741667151451111 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873698949813843 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7525941133499146 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.038845330476760864 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933527112007141 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7503779530525208 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21294446289539337 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20323598384857178 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4554491937160492 *jet/accuracy_1_of_2‚0.8098947405815125 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25253716111183167 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.726297914981842 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6697901487350464 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 &particle/f_score‚0.651530921459198 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03850415349006653 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 )loss/val_total_loss‚8.702550490651694 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7116292119026184 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3056311011314392HÕ¬=Õ¦{ :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 % _timestamp‚1.7843605980898123e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7594316601753235 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6110759377479553 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2019769847393036 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9929133653640747 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6195901036262512 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18583333492279053 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8473358154296875 particle/accuracy_1_of_4‚1 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.98603355884552 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3122864365577698 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7323473691940308 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9953984022140503 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24564072489738464 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9953984022140503 +jet/accuracy_2_of_2‚0.15570756793022156 +jet/accuracy_3_of_4‚0.09999000281095505 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5572516322135925 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26383528113365173 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24542947113513947 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4849550127983093 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3599 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7430240511894226 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7102619409561157 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6763437986373901 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6812182068824768 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19197121262550354 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9925982356071472 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2640424966812134 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7583575248718262 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04075905308127403 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9894558191299438 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22943246364593506 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9859439134597778 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13742682337760925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20802223682403564 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8473358154296875 +jet/accuracy_3_of_3‚0.07372218370437622 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20702488720417023 _step‚296 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5520271062850952 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955244660377502 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9265109300613403 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12201228737831116 *jet/accuracy_2_of_4‚0.8141986131668091 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936091899871826 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6208056211471558 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0916481539607048 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.666715681552887 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6623740196228027 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2249438613653183 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 epoch‚8 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03939174860715866 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6743825078010559 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0423104465007782 particle/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860796332359314 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4554497003555298 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22502022981643677 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.895575761795044 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7549886703491211 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7665966749191284 *jet/accuracy_1_of_4‚0.9682220220565796 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2642631530761719 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4686416983604431 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8768740892410278 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3957784175872803 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995604395866394 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7618318796157837 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.550156831741333 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4427083432674408 _runtime‚3090.165811739 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6697901487350464 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6338476538658142 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.835355281829834 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07586878538131714 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04107978567481041 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8607978820800781 ,jet/accuracy_4_of_4‚0.013899499550461769 6validation_average_jet_accuracy‚0.3877866864204407 particle/accuracy_3_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24835191667079926 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2530174255371094 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9327751398086548 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047986030578613 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6822916865348816 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04982021823525429 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7630487084388733 *jet/accuracy_2_of_3‚0.5696191787719727 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7618455290794373 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8280045390129089 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6675854921340942 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6152611970901489 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8450114130973816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6892662048339844 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7076148986816406 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_3_of_4‚1 particle/accuracy_4_of_5‚1 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992159366607666 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7151239514350891 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39753296971321106 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6484599709510803 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9103319048881531 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.650034487247467 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962165355682373 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44615691900253296 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8017535209655762 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2652738690376282 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.996662437915802 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7270690202713013 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7634421586990356 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047763172537088394 .particle/accuracy_4_of_4‚0.954102098941803 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995124101638794 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2530174255371094 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039602428674697876 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0423104465007782 particle/accuracy_2_of_3‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47048649191856384 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8223734498023987 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5741667151451111 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873698949813843 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7525941133499146 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.038845330476760864 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933527112007141 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7503779530525208 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21294446289539337 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20323598384857178 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4554491937160492 *jet/accuracy_1_of_2‚0.8098947405815125 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25253716111183167 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.726297914981842 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6697901487350464 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 &particle/f_score‚0.651530921459198 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03850415349006653 jet/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 )loss/val_total_loss‚8.702550490651694 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7116292119026184 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3056311011314392 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1535269021987915 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6713477373123169 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8441988825798035 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22544443607330322 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3405666649341583 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9388644695281982 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7"Purity/1FRt1SRqqt1FBt/event_purity‚0.30634805560112 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/sensitivity‚0.6416136026382446 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9944863319396973 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21352066099643707 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13743001222610474 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_1‚0.3471159338951111 (particle/accuracy‚0.8526204228401184 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0¨¥>œÖ‚(:‘ ¤ÝìÒ«°Â» gpu.0.gpu‚16.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾8Öéׂ(:Þ ¤ÝìÒû¨Â¼' disk.nvme0c0n1.out‚11171.3916015625$ memory_percent‚95.46747207641602* disk./.usagePercent‚39.526066154640674 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 247.734375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 5221.28515625 cpu‚26.195118364612586 disk.sda1.in‚0 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0% disk.nvme0n1.out‚11171.3916015625 proc.cpu.threads‚17 disk.sda.out‚ 247.734375$ disk./.usageGB‚588.9669494628906 disk.sda1.out‚0* proc.memory.percent‚16.920852661132812 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 247.734375 network.recv‚848578 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 disk.dm-0.in‚ 5221.28515625 disk.dm-1.in‚ 0.00390625! disk.nvme0n1.in‚ 5221.28515625 network.sent‚ 5.627792e+06 disk.drbd2165.out‚0" disk.dm-0.out‚11171.3916015625 disk.rbd1.out‚91.61962890625& proc.memory.availableMB‚ 742.609375 proc.memory.rssMB‚ 2772.31257,“ù“Ø‚(:ˆ ³ÝìÒÊÊñ» gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚242.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.5342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“×åÙ‚(:Ú ³ÝìÒɲë½$ disk./.usageGB‚588.9661407470703 disk.sda1.out‚0$ memory_percent‚95.46875953674316% disk.nvme0c0n1.out‚11421.45703125 disk.dm-1.in‚ 0.00390625 proc.cpu.threads‚17 network.sent‚ 5.685146e+06 network.recv‚853098 disk.rbd1.out‚91.61962890625 disk.rbd0.out‚0 disk.sda.in‚ 0.00390625 disk.sda.out‚ 248.1640625 disk.dm-0.in‚ 5221.28515625 disk.dm-1.out‚ 248.1640625* proc.memory.percent‚16.920852661132812 cpu‚26.19432381150927 disk.sda3.out‚ 248.1640625 disk.sda1.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2772.3125) disk./.usagePercent‚39.52601188104342 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 5221.28515625# disk.nvme0n1.out‚11421.45703125 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 742.3984375 disk.rbd1.in‚ 0.01953125! disk.nvme0n1.in‚ 5221.28515625 disk.sda3.in‚ 0.00390625 disk.dm-0.out‚11421.45703125j¥ 2Ú- _wandbruntime‚3132  _runtime‚3132KºMäÆÛÀ _step‚297 & lr-AdamW/pg1‚0.0004061675088110701 & lr-AdamW/pg2‚0.0004061675088110701 trainer/global_step‚3649 % _timestamp‚1.7843606405737898e+09 _runtime‚ 3132.64911221‹ n#ËÜÅ trainer/global_step‚3649 % _timestamp‚1.7843606405737898e+09 _runtime‚ 3132.64911221 _step‚297 & lr-AdamW/pg1‚0.0004061675088110701 & lr-AdamW/pg2‚0.0004061675088110701©‘dm2Ý- _wandbruntime‚3132  _runtime‚3132§6”à… ÞÚO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554831698993152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14245690 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554831681953792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558693001809920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751368 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3903704905728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554831698993152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261862 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚3649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126185752022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3903614231552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7847594 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122282156402688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643431424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643431424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62761347 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚551977188036608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDœü¤o[eviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3860961477632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚551977205075968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28946656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3861302816768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6397811 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3132.650375345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122282137790464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚555832259748580 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508`À»ûå'ßßO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚558693001809920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3860961477632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 _step‚298 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3903704905728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554831698993152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261862 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚3649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126185752022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3903614231552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7847594 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122282156402688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643431424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643431424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62761347 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚551977188036608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3860961477632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚551977205075968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28946656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3861302816768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6397811 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3132.650375345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122282137790464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚555832259748580 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554831698993152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14245690 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554831681953792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558693001809920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751368 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554978940022784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28954286 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3904647095808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _step‚299 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33823757 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3904731471360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚552123698708480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778045 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14249146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33813766 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚558841255473664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33823757 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554978922983424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3856082765694 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126220316632576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122315585161216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28954286 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255473664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555980139063536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554978940022784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315450880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6399556 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62778045 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚558840897088000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ñ ÌÎï&âéM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554978940022784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28954286 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3904647095808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _step‚299 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33823757 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3904731471360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚552123698708480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778045 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14249146 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33813766 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚558841255473664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33823757 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554978922983424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3856082765694 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126220316632576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122315585161216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28954286 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255473664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555980139063536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554978940022784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315450880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6399556 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62778045 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚558840897088000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390414848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29328254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843606813946185e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29328254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29328256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3911998318592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3954988584960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587420 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748793344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597399 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562193750474752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34259166 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6485058 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127908665199616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911998318592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚566105748793344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911998318592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚566105748793344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390414848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29328254 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843606813946185e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29328254 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29328256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3911998318592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3954988584960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587420 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748793344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597399 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438®;\2ò- _wandbruntime‚3174  _runtime‚3174,•ƒ®oóäM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3912669606400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚559449232302080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562340974465024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3956094007296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14440877 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562340991504384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123987004475392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6486803 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34268218 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278209 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3956007534592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6486806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127943117094912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚563355931554682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562340991504384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123987023087616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843606820109713e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127943012009984 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚566253644071424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚303 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566254002457088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29335886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚3699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34268218 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14440604 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3912669606400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7954071 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335886 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566254002457088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3906699252602 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3907039802604 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3913010952704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562340974465024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34278209 _runtime‚3174.086546133 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614097 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14440604 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3912669606400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7954071 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335886 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566254002457088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3906699252602 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3907039802604 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3913010952704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562340974465024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34278209 _runtime‚3174.086546133 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614097 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚559449232302080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562340974465024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3956094007296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14440877 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562340991504384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123987004475392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6486803 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34268218 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278209 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3956007534592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6486806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127943117094912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚563355931554682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562340991504384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2¯kNQF2õ- _wandbruntime‚3174  _runtime‚3174¢—øºö 0loss/FRt1/assignment_loss‚1.7483110715486776 epoch‚9 _runtime‚3174.088536531 /loss/FBt1/detection_loss‚1.5525099569931626 %loss/total_loss‚9.689406887417261 2loss/SRqqt2/assignment_loss‚0.5438555413470391 .loss/FRt1/detection_loss‚0.551547052955427 1loss/SRqqt1/detection_loss‚1.0825255376879463 1loss/SRqqt2/detection_loss‚0.6403284106724942 % _timestamp‚1.7843606820134573e+09 /loss/FBt2/assignment_loss‚0.038748271681025 trainer/global_step‚3699 /loss/FBt2/detection_loss‚0.6645603665313214 0loss/FRt2/assignment_loss‚0.7012711847063355 2loss/SRqqt1/assignment_loss‚0.6087015916118808 4loss/FBt1/assignment_loss‚0.00019358081069853483 /loss/FRt2/detection_loss‚1.5568543208712537 _step‚304>h›B ÷† trainer/global_step‚3699 /loss/FBt2/detection_loss‚0.6645603665313214 0loss/FRt2/assignment_loss‚0.7012711847063355 2loss/SRqqt1/assignment_loss‚0.6087015916118808 4loss/FBt1/assignment_loss‚0.00019358081069853483 /loss/FRt2/detection_loss‚1.5568543208712537 _step‚304 0loss/FRt1/assignment_loss‚1.7483110715486776 epoch‚9 _runtime‚3174.088536531 /loss/FBt1/detection_loss‚1.5525099569931626 %loss/total_loss‚9.689406887417261 2loss/SRqqt2/assignment_loss‚0.5438555413470391 .loss/FRt1/detection_loss‚0.551547052955427 1loss/SRqqt1/detection_loss‚1.0825255376879463 1loss/SRqqt2/detection_loss‚0.6403284106724942 % _timestamp‚1.7843606820134573e+09 /loss/FBt2/assignment_loss‚0.038748271681025°‚‰zt“ø‚(:ˆ ïÝìÒû´» gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xmÐãù‚(:Ø ïÝìÒ¾¼Å½ disk.rbd0.out‚0 disk.sda3.out‚ 249.578125 disk.sda1.out‚0# disk./.usageGB‚588.965877532959 disk.sda2.out‚0 disk.dm-0.in‚ 5289.6640625 disk.rbd1.in‚ 0.01953125 disk.drbd2165.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 5289.6640625 disk.rbd1.out‚91.76025390625 disk.sda2.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.920852661132812" disk.dm-0.out‚11440.7822265625 disk.sda3.in‚ 0.00390625% proc.memory.availableMB‚ 495.90625 disk.dm-1.in‚ 0.00390625 disk.nvme0n1.in‚ 5289.6640625 proc.memory.rssMB‚ 2772.3125 network.sent‚ 5.761944e+06) disk./.usagePercent‚39.52599421652355 disk.dm-1.out‚ 249.578125% disk.nvme0n1.out‚11440.7822265625 disk.sda.out‚ 249.578125$ memory_percent‚96.97322845458984 proc.cpu.threads‚17 network.recv‚866279 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚11440.7822265625 disk.sda.in‚ 0.00390625 cpu‚26.191080059284367<Ÿœ“ú‚(:ˆ þÝìÒë´» gpu.0.gpu‚42.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W©W@êû‚(:ß þÝìÒŽËî¼ proc.memory.rssMB‚ 2772.3125 disk.sda2.out‚0 disk.dm-1.out‚ 249.98828125! disk.nvme0c0n1.in‚ 5359.390625 cpu‚26.190279224371356) disk./.usagePercent‚39.52599882465917 disk.nvme0n1.in‚ 5359.390625 disk.sda2.in‚0 disk.dm-0.in‚ 5359.390625" disk.dm-0.out‚11444.2373046875 disk.rbd1.out‚91.76025390625 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚870667% disk.nvme0n1.out‚11444.2373046875 disk.rbd0.out‚0 network.sent‚ 5.824862e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.920852661132812$ disk./.usageGB‚588.9659461975098 disk.sda.in‚ 0.00390625 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01953125' disk.nvme0c0n1.out‚11444.2373046875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 249.98828125$ memory_percent‚96.97682857513428 disk.sda.out‚ 249.98828125 disk.dm-1.in‚ 0.00390625( proc.memory.availableMB‚ 495.31640625. õ“ü‚(:ˆ ÞìÒ•šª¼ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚259.411' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½î+Oßý‚(:Ô ÞìÒøû’½ disk.rbd0.out‚0% disk.nvme0c0n1.out‚11450.03515625 disk.sda.out‚ 250.328125 disk.sda1.out‚0* proc.memory.percent‚16.920852661132812 disk.sda.in‚ 0.00390625 proc.cpu.threads‚17 disk.dm-1.out‚ 250.328125 disk.rbd1.out‚91.90087890625 disk.drbd2165.in‚0 cpu‚26.189237129980516 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 5364.8359375 disk.sda2.out‚0# disk./.usageGB‚588.965892791748 disk.sda1.in‚0 disk.drbd2165.out‚0 network.sent‚ 5.82994e+06) disk./.usagePercent‚39.52599524055369 disk.nvme0n1.in‚ 5364.8359375 disk.dm-0.in‚ 5364.8359375( proc.memory.availableMB‚ 495.26171875 proc.memory.rssMB‚ 2772.3125# disk.nvme0n1.out‚11450.03515625 disk.sda3.out‚ 250.328125$ memory_percent‚96.97716236114502 network.recv‚874099 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.dm-0.out‚11450.03515625Xåv62þ- _wandbruntime‚3214  _runtime‚3214N[“›Éÿà ' lr-AdamW/pg2‚0.00040440880206736736 trainer/global_step‚3749 % _timestamp‚1.7843607222937887e+09 _runtime‚3214.369087473 _step‚305 ' lr-AdamW/pg1‚0.00040440880206736736 >h~Î€È ' lr-AdamW/pg1‚0.00040440880206736736 ' lr-AdamW/pg2‚0.00040440880206736736 trainer/global_step‚3749 % _timestamp‚1.7843607222937887e+09 _runtime‚3214.369087473 _step‚305±¼—Í2- _wandbruntime‚3214  _runtime‚3214gÑ•7à'‚ÚO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29709856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29709854 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚3214.373093746 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚569555784916992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34713618 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3962693820416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129631432794624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29709856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6572311 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573518495776768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3962693820416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚3749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34723595 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6572308 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34723595 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚569555801956352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8056123 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137398272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4006305670656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3956645235678 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚569555784916992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129631319313920 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14628180 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4006400539136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433451 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14628434 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34713618 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚570584917498846 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576t*ጚ ƒßO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137398272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4006305670656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3956645235678 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚569555784916992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129631319313920 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14628180 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4006400539136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433451 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14628434 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34713618 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚570584917498846 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29709856 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29709854 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚3214.373093746 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚569555784916992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34713618 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3962693820416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129631432794624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29709856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6572311 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573518495776768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3962693820416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚3749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34723595 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6572308 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34723595 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚569555801956352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8056123 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979² %<2„- _wandbruntime‚3214  _runtime‚3214s#©ê&…äM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566774782935040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129665887162368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3957656289512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963706454528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3214.987320175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963365108224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129665782077440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29717488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569703025946624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4007339675648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29717488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚570732439224552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732661 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8057856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4007426148352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573666749440512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957315739510 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573666749440512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125658442401792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34722670 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440156 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569703042985984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14631909 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8058131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843607229115515e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440156 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34722670 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14631909 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8058131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843607229115515e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440156 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34722670 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3214.987320175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963365108224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129665782077440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29717488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569703025946624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4007339675648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29717488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚570732439224552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732661 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8057856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4007426148352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573666749440512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957315739510 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573666749440512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125658442401792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34722670 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440156 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888³xR¥X2‡- _wandbruntime‚3214  _runtime‚3214/BŽ}ˆÿ 1loss/SRqqt1/detection_loss‚0.9081707262149035 %loss/total_loss‚8.987054652793836 /loss/FRt1/assignment_loss‚1.875146873795772 .loss/FRt2/detection_loss‚1.576498930415828 1loss/SRqqt2/detection_loss‚0.6370103667677742 0loss/FBt2/detection_loss‚0.47855172735888146 _runtime‚3214.989608805 _step‚308 1loss/SRqqt1/assignment_loss‚0.652408080881986 0loss/FBt1/assignment_loss‚0.0651686362963801 epoch‚9 trainer/global_step‚3749 3loss/SRqqt2/assignment_loss‚0.48811198128564176 1loss/FBt2/assignment_loss‚0.05882555686690619 /loss/FRt1/detection_loss‚0.5001056977408478 /loss/FBt1/detection_loss‚1.0517229779313009 % _timestamp‚1.7843607229145296e+09 0loss/FRt2/assignment_loss‚0.6953330972376145YÈ}ç ‰„ /loss/FRt1/assignment_loss‚1.875146873795772 .loss/FRt2/detection_loss‚1.576498930415828 1loss/SRqqt2/detection_loss‚0.6370103667677742 0loss/FBt2/detection_loss‚0.47855172735888146 _runtime‚3214.989608805 _step‚308 1loss/SRqqt1/assignment_loss‚0.652408080881986 0loss/FBt1/assignment_loss‚0.0651686362963801 epoch‚9 trainer/global_step‚3749 3loss/SRqqt2/assignment_loss‚0.48811198128564176 1loss/FBt2/assignment_loss‚0.05882555686690619 /loss/FRt1/detection_loss‚0.5001056977408478 /loss/FBt1/detection_loss‚1.0517229779313009 % _timestamp‚1.7843607229145296e+09 0loss/FRt2/assignment_loss‚0.6953330972376145 1loss/SRqqt1/detection_loss‚0.9081707262149035 %loss/total_loss‚8.987054652793836´ÚFäΞŠ‚(:“ œÞìÒߎ¿» gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N†–rë‹‚(:à œÞìÒŒúм disk.sda2.out‚0( proc.memory.availableMB‚ 494.73046875 cpu‚26.18860306511004 disk.drbd2165.out‚0 disk.rbd0.out‚0# memory_percent‚96.9804048538208 disk.drbd2165.in‚0 disk.dm-1.out‚ 250.79296875 disk.sda3.out‚ 250.79296875 disk.sda1.out‚0% disk.nvme0n1.out‚11453.4580078125 network.sent‚ 5.834844e+06 disk.dm-0.in‚ 5364.8359375 disk.rbd1.out‚91.90087890625" disk.nvme0c0n1.in‚ 5364.8359375 disk.nvme0n1.in‚ 5364.8359375 disk.sda1.in‚0 proc.memory.rssMB‚ 2772.3125" disk.dm-0.out‚11453.4580078125 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.sda.out‚ 250.79296875* proc.memory.percent‚16.920852661132812$ disk./.usageGB‚588.9659080505371 disk.rbd1.in‚ 0.01953125 proc.cpu.threads‚17 network.recv‚877066) disk./.usagePercent‚39.52599626458383' disk.nvme0c0n1.out‚11453.4580078125 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0Î^”æ‘Œ‚(:† «ÞìÒ‘¹ç» gpu.0.gpu‚41.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ]Ø_ß‚(:Ô «ÞìÒƶ¼¼* disk./.usagePercent‚39.525987560327664$ disk./.usageGB‚588.9657783508301 disk.dm-0.out‚11460.71484375 disk.rbd0.in‚0% disk.nvme0c0n1.out‚11460.71484375 disk.sda1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 494.7109375" disk.nvme0c0n1.in‚ 5364.8359375 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚881484 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚91.96337890625 disk.sda3.out‚ 251.109375 cpu‚26.1878159056449 disk.rbd1.in‚ 0.01953125# disk.nvme0n1.out‚11460.71484375$ memory_percent‚96.98052406311035 proc.cpu.threads‚17 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 251.109375 disk.sda.out‚ 251.109375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 5364.8359375* proc.memory.percent‚16.920852661132812 network.sent‚ 5.899759e+06 disk.dm-0.in‚ 5364.8359375 disk.sda.in‚ 0.00390625 proc.memory.rssMB‚ 2772.3125«/pZ‘Ž‚(:† ºÞìÒÇþ·¼ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!DüW„‚(:× ºÞìÒ…ǽ disk.sda1.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚11465.3916015625 disk.dm-1.in‚ 0.00390625 network.recv‚884981" disk.nvme0c0n1.in‚ 5364.8359375' disk.nvme0c0n1.out‚11465.3916015625 disk.sda3.out‚ 251.640625 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 di‘ÛüÜ^sk.dm-0.in‚ 5364.8359375 network.sent‚ 5.902921e+06( disk./.usagePercent‚39.5259967765989$ disk./.usageGB‚588.9659156799316 disk.rbd0.in‚0% disk.nvme0n1.out‚11465.3916015625 disk.sda.out‚ 251.640625 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2772.3125 disk.sda1.out‚0$ memory_percent‚96.97914123535156$ proc.memory.availableMB‚494.9375 disk.dm-1.out‚ 251.640625 cpu‚26.186481471364505 disk.rbd1.out‚91.96337890625* proc.memory.percent‚16.920852661132812 disk.rbd1.in‚ 0.01953125 disk.nvme0n1.in‚ 5364.8359375 disk.sda.in‚ 0.00390625‘……¾2- _wandbruntime‚3255  _runtime‚3255õCÒ•É‘Ã ' lr-AdamW/pg2‚0.00040351141304680694 trainer/global_step‚3799 % _timestamp‚1.7843607630829065e+09 _runtime‚3255.158591305 _step‚309 ' lr-AdamW/pg1‚0.00040351141304680694Fð8®Î’È _step‚309 ' lr-AdamW/pg1‚0.00040351141304680694 ' lr-AdamW/pg2‚0.00040351141304680694 trainer/global_step‚3799 % _timestamp‚1.7843607630829065e+09 _runtime‚3255.158591305µåòô2“- _wandbruntime‚3255  _runtime‚3255&çCß'”ÙO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576917853437952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131354133689856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389322240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884381696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168070 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30091456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884381696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4057572833792 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990104 trainer/global_step‚3799 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269503 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576917853437952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6659558 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577961067579354 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784360763084183e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30091456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35178047 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14819330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242760192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35178047 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8159512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4007261722586 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127296451569664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14819070Õº…ä'•ÞO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6659561 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30091454 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30091454 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580931242760192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4013047983104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4057663507968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013389322240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35178047 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8159512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4007261722586 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168070 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127296451569664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14819070 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576917853437952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131354133689856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389322240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884381696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168070 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30091456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884381696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4057572833792 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990104 trainer/global_step‚3799 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269503 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576917853437952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6659558 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577961067579354 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784360763084183e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30091456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35178047 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14819330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242760192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276208 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286201 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14823119 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35187113 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577065077428224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4014060610048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131388581399552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138038272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8161552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14822855 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131388472120320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286201 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127329898940416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4058591792128 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30099088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4058682459136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30099086 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401956352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577065077428224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8161813 _step‚311n¤ä¼ƒ˜èM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577065077428224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8161813 _step‚311 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4014401956352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚574100299489280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6661306 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 GÒ¢8k"_runtime‚ 3255.77802784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578108231715698 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276208 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286201 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14823119 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35187113 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577065077428224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4014060610048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131388581399552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138038272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8161552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14822855 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131388472120320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286201 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127329898940416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4058591792128 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30099088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4058682459136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30099086 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401956352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0·œÇ:2™- _wandbruntime‚3255  _runtime‚3255Ïj(Åš‚ /loss/FRt2/detection_loss‚1.5241591176134826 0loss/FRt2/assignment_loss‚0.6768489598032824 1loss/SRqqt1/detection_loss‚0.8603528871924596 _step‚312 /loss/FBt1/detection_loss‚0.8920267144946948 0loss/FBt2/detection_loss‚0.46127749101861437 %loss/total_loss‚8.578367507100623 epoch‚9 trainer/global_step‚3799 % _timestamp‚1.7843607637040322e+09 _runtime‚ 3255.77915076 0loss/FRt1/assignment_loss‚1.8504182169353633 2loss/SRqqt1/assignment_loss‚0.5075518054470567 2loss/SRqqt2/assignment_loss‚0.5209559459093644 2loss/FBt2/assignment_loss‚0.046781522888968814 1loss/SRqqt2/detection_loss‚0.6926182978546359 1loss/FBt1/assignment_loss‚0.00543351358268751 /loss/FRt1/detection_loss‚0.5399430343600133ÖÑ“Z ›‡ 0loss/FBt2/detection_loss‚0.46127749101861437 %loss/total_loss‚8.578367507100623 epoch‚9 trainer/global_step‚3799 % _timestamp‚1.7843607637040322e+09 _runtime‚ 3255.77915076 0loss/FRt1/assignment_loss‚1.8504182169353633 2loss/SRqqt1/assignment_loss‚0.5075518054470567 2loss/SRqqt2/assignment_loss‚0.5209559459093644 2loss/FBt2/assignment_loss‚0.046781522888968814 1loss/SRqqt2/detection_loss‚0.6926182978546359 1loss/FBt1/assignment_loss‚0.00543351358268751 /loss/FRt1/detection_loss‚0.5399430343600133 /loss/FRt2/detection_loss‚1.5241591176134826 0loss/FRt2/assignment_loss‚0.6768489598032824 1loss/SRqqt1/detection_loss‚0.8603528871924596 _step‚312 /loss/FBt1/detection_loss‚0.8920267144946948¸S4¼¼“œ‚(:ˆ ÉÞìÒ·£Øº gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ætñ‚(:æ ÉÞìҘߌ½& disk.nvme0n1.out‚11471.91650390625 disk.sda3.out‚ 251.8046875 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚11471.91650390625 network.sent‚ 5.907886e+06 network.recv‚888140$ disk./.usageGB‚588.9654998779297 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11471.91650390625 disk.sda.out‚ 251.8046875 disk.dm-0.in‚ 5424.80859375 disk.dm-1.out‚ 251.8046875 proc.memory.rssMB‚ 2772.3125 cpu‚26.185563749164118 disk.drbd2165.out‚0* disk./.usagePercent‚39.525968871777664 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 5424.80859375 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.drbd2165.in‚0$ memory_percent‚96.97916507720947* proc.memory.percent‚16.920852661132812 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.sda2.out‚0( proc.memory.availableMB‚ 494.93359375 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚92.04150390625! disk.nvme0n1.in‚ 5424.80859375)6Æ œž‚(:‘ ØÞìÒ¯ù⺠gpu.0.gpu‚33.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.27' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.854000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)IC(:ä ØÞìÒÄäÛ» network.sent‚ 5.970609e+06 disk.sda3.out‚ 252.25390625 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 proc.memory.rssMB‚ 2772.3125) disk./.usagePercent‚39.52596656770986& disk.nvme0n1.out‚11475.73095703125 disk.sda3.in‚ 0.00390625# disk.dm-0.out‚11475.73095703125$ memory_percent‚96.97933197021484$ disk./.usageGB‚588.9654655456543 disk.sda.out‚ 252.25390625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 5424.80859375 disk.rbd1.out‚92.04150390625* proc.memory.percent‚16.920852661132812 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5424.80859375 disk.sda.in‚ 0.00390625 disk.dm-1.out‚ 252.25390625% proc.memory.availableMB‚ 494.90625 network.recv‚892497( disk.nvme0c0n1.out‚11475.73095703125 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 5424.80859375 disk.sda2.in‚0 disk.rbd0.out‚0 cpu‚26.18512340607667 proc.cpu.threads‚17æ¹*a2 - _wandbruntime‚3296  _runtime‚3296"øb´È¡Â ' lr-AdamW/pg1‚0.00040260207685358486 ' lr-AdamW/pg2‚0.00040260207685358486 trainer/global_step‚3849 $ _timestamp‚1.784360804006793e+09 _runtime‚3296.082124002 _step‚313¹J²Í¢Ç ' lr-AdamW/pg1‚0.00040260207685358486 ' lr-AdamW/pg2‚0.00040260207685358486 trainer/global_step‚3849 $ _timestamp‚1.784360804006793e+09 _runtime‚3296.082124002 _step‚313¹’γ+2£- _wandbruntime‚3296  _runtime‚3296, &à'¤ÚO _runtime‚3296.083082201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30473056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚314 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128967889496064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585337217659862 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631365120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4064084824064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66105555 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128967908108288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4064084824064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30473054 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4063743484928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35632499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622522 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588343989743616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15010352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4108941339136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35632499 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584279887880192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8263295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584279887880192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4063743484928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133076849447424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8263541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133076740161024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚3849 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4057537659604 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584279904919552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4057878209494ÓÈÝ!"¥ßO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4064084824064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30473054 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4063743484928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35632499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622522 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚588343989743616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15010352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4108941339136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35632499 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584279887880192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8263295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584279887880192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4063743484928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133076849447424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8263541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133076740161024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚3849 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4057537659604 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584279904919552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4057878209494 _runtime‚3296.083082201 QIDeviceStats¸ZÔáMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30473056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚314 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128967889496064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585337217659862 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631365120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4064084824064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66105555 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128967908108288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ºWô2¦- _wandbruntime‚3296  _runtime‚3296ÍBD‰é&§ãM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15013782 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584427128909824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35641565 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8265229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6748556 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30480688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4058548713326 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584427145949184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚585484739385568 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122253 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15014051 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129001318254592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚581425850122240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756111872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243407360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35631574 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097458176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641565 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122253 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 40647561118726´”î&¨èM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122253 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4064756111872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15013782 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584427128909824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35641565 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8265229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6748556 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30480688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4058548713326 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584427145949184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚585484739385568 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122253 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15014051 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129001318254592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚581425850122240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756111872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243407360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35631574 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097458176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641565 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269»ëû•O2©- _wandbruntime‚3296  _runtime‚3296A伪€ 2loss/FBt2/assignment_loss‚0.012923192674805352 epoch‚9 % _timestamp‚1.7843608046262214e+09 1loss/SRqqt2/detection_loss‚0.6111709792925849 .loss/FBt1/detection_loss‚0.762663087752812 trainer/global_step‚3849 0loss/FRt2/assignment_loss‚0.7335526334626652 2loss/SRqqt1/assignment_loss‚0.6682485232261737 .loss/FRt2/detection_loss‚1.569422824992624 1loss/SRqqt1/detection_loss‚0.9425316296212399 /loss/FBt2/detection_loss‚0.4204910699205071 _runtime‚3296.701489983 0loss/FRt1/assignment_loss‚1.6816232412441734 2loss/SRqqt2/assignment_loss‚0.5750075413513053 /loss/FRt1/detection_loss‚0.5799918795395888 %loss/total_loss‚8.574747865816594 _step‚316 1loss/FBt1/assignment_loss‚0.01712126273811437¸úX@ «… _step‚316 1loss/FBt1/assignment_loss‚0.01712126273811437 2loss/FBt2/assignment_loss‚0.012923192674805352 epoch‚9 % _timestamp‚1.7843608046262214e+09 1loss/SRqqt2/detection_loss‚0.6111709792925849 .loss/FBt1/detection_loss‚0.762663087752812 trainer/global_step‚3849 0loss/FRt2/assignment_loss‚0.7335526334626652 2loss/SRqqt1/assignment_loss‚0.6682485232261737 .loss/FRt2/detection_loss‚1.569422824992624 1loss/SRqqt1/detection_loss‚0.9425316296212399 /loss/FBt2/detection_loss‚0.4204910699205071 _runtime‚3296.701489983 0loss/FRt1/assignment_loss‚1.6816232412441734 2loss/SRqqt2/assignment_loss‚0.5750075413513053 /loss/FRt1/detection_loss‚0.5799918795395888 %loss/total_loss‚8.574747865816594¼dEG.ꬂ(:ß çÞìÒ˜²Ñ¼ disk.rbd1.out‚92.10400390625# disk.nvme0c0n1.in‚ 5424.80859375 network.recv‚896196 disk.dm-0.in‚ 5424.80859375" disk.dm-0.out‚11480.6240234375 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0$ disk./.usageGB‚588.9649047851562 disk.drbd2165.in‚0% disk.nvme0n1.out‚11480.6240234375$ memory_percent‚96.97399139404297* proc.memory.percent‚16.920852661132812 cpu‚26.184122373244612 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 252.7734375% proc.memory.availableMB‚ 495.78125) disk./.usagePercent‚39.52592893460232 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 5424.80859375 disk.sda.in‚ 0.00390625 proc.memory.rssMB‚ 2772.3125 disk.dm-1.out‚ 252.7734375' disk.nvme0c0n1.out‚11480.6240234375 disk.sda.out‚ 252.7734375 disk.sda3.in‚ 0.00390625 proc.cpu.threads‚17 network.sent‚ 5.975943e+06 disk.rbd1.in‚ 0.01953125ðÎ2“­‚(:ˆ çÞìÒÊîæ¼ gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚238.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k$Ç–“®‚(:ˆ öÞìÒò„×» gpu.0.gpu‚44.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹§µ6鯂(:Þ öÞìҚо½ disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0! disk.nvme0n1.in‚ 5424.80859375 cpu‚26.18343126279666 proc.cpu.threads‚17 network.recv‚900211* disk./.usagePercent‚39.525928422587256 disk.sda1.out‚0 disk.sda2.out‚0$ proc.memory.availableMB‚495.5625 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2772.3125* proc.memory.percent‚16.920852661132812 disk.sda.out‚ 253.140625 disk.sda3.out‚ 253.140625 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 253.140625$ disk./.usageGB‚588.9648971557617 disk.sda2.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚11483.84326171875 disk.rbd1.out‚92.10400390625 disk.rbd0.out‚0& disk.nvme0n1.out‚11483.84326171875 network.sent‚ 6.038611e+06# disk.dm-0.out‚11483.84326171875# disk.nvme0c0n1.in‚ 5424.80859375$ memory_percent‚96.97532653808594 disk.sda3.in‚ 0.00390625 disk.dm-0.in‚ 5424.80859375Œ lî“°‚(:ˆ …ßìÒ•·Ž» gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôì¿Ü±‚(:Ñ …ßìÒÜ ¼ disk.sda2.in‚0 disk.dm-0.in‚ 5488.890625# memory_percent‚96.9738245010376( proc.memory.availableMB‚ 495.80859375 disk.rbd1.out‚92.18212890625 disk.sda.out‚ 253.34375 disk.sda3.out‚ 253.34375) disk./.usagePercent‚39.52593763885849 disk.rbd1.in‚ 0.01953125 disk.nvme0n1.in‚ 5488.890625 disk.drbd2165.out‚0! disk.dm-0.out‚11488.806640625 disk.sda2.out‚0 proc.memory.rssMB‚ 2772.3125 network.sent‚ 6.04358e+06 disk.drbd2165.in‚0 cpu‚26.182258382790856$ disk./.usageGB‚588.9650344848633 disk.sda1.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 5488.890625 disk.sda.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚11488.806640625 disk.sda3.in‚ 0.00390625* proc.memory.percent‚16.920852661132812 proc.cpu.threads‚17$ disk.nvme0n1.out‚11488.806640625 network.recv‚903908 disk.dm-1.out‚ 253.34375 disk.sda1.in‚0ZØ2²- _wandbruntime‚3337  _runtime‚3337n(dzÁ & lr-AdamW/pg2‚0.0004016808518926655 trainer/global_step‚3899 % _timestamp‚1.7843608450108278e+09 _runtime‚3337.089214304 _step‚317 & lr-AdamW/pg1‚0.0004016808518926655”ÿ?Ì´Æ & lr-AdamW/pg1‚0.0004016808518926655 & lr-AdamW/pg2‚0.0004016808518926655 trainer/global_step‚3899 % _timestamp‚1.7843608450108278e+09 _runtime‚3337.089214304 _step‚317½¡m2µ- _wandbruntime‚3337  _runtime‚3337=,Óèö¶ÚO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚591641956401152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8367164 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6834061 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚3899 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚591641956401152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114780325888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8367428 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086951 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36086951 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30854656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚595756378348544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30854656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134799378233856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚595756736727040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚318 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134799487520256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚592713367740370 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚3337.090204283 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588604873043968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4114438986752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚¯³ê9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36076974 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15201489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15201222 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30854654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4108154146512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4114438986752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4160050811392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592713010151120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4160141485568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4114780325888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚595756736727040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130639346034688 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378348544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588604856004608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚591641939361792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8367428 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086951 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36086951 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30854656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚595756378348544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30854656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134799378233856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚595756736727040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚318 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134799487520256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚592713367740370 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚3337.090204283 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588604873043968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4114438986752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36076974 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15201489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15201222 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30854654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4108154146512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4114438986752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4160050811392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592713010151120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4160141485568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4114780325888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚595756736727040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130639346034688 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378348544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588604856004608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚591641939361792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6834061 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚3899 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚591641956401152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114780325888¾#í!g2¸- _wandbruntime‚3337  _runtime‚3337r ½Ãê&¹äM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30862288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4161158834176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4109165200234 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30862286 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30862288 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚591789180391424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4161066070016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36096017 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15205010 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134833822251008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843608456297162e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6835803 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8369207 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4109505750236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592860889466076 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚595904990390784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588751366676480 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30862286 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8369479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134833933627392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4115451613696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚3337.715845833 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66958305 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚595904990390784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115792960000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36086026 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115792960000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130672774793216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588751383715840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958305 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36086026 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115792960000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130672774793216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588751383715840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958305 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30862288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4161158834176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4109165200234 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30862286 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30862288 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_endùß·¡ý /allocated_bytes.large_pool.freed‚591789180391424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4161066070016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36096017 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15205010 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134833822251008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843608456297162e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6835803 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8369207 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¿Wš¸-2»- _wandbruntime‚3337  _runtime‚3337Ê·Ô\¼€ epoch‚9 /loss/FRt1/detection_loss‚0.5479411634666017 /loss/FRt2/detection_loss‚1.6171979749790177 0loss/FBt2/detection_loss‚0.45677509264964045 2loss/SRqqt1/assignment_loss‚0.5311129717602118 .loss/FBt1/detection_loss‚1.121089223171434 _runtime‚ 3337.71656991 0loss/FRt1/assignment_loss‚1.8444998278864113 0loss/FRt2/assignment_loss‚0.7520545551410088 1loss/FBt1/assignment_loss‚0.12734882790836022 2loss/FBt2/assignment_loss‚0.048065646870011074 0loss/SRqqt2/detection_loss‚0.701271556686961 trainer/global_step‚3899 % _timestamp‚1.7843608456414433e+09 _step‚320 2loss/SRqqt2/assignment_loss‚0.5314214710973645 1loss/SRqqt1/detection_loss‚0.9832669417878055 %loss/total_loss‚9.262045253404828IJSõ ½… 2loss/SRqqt2/assignment_loss‚0.5314214710973645 1loss/SRqqt1/detection_loss‚0.9832669417878055 %loss/total_loss‚9.262045253404828 epoch‚9 /loss/FRt1/detection_loss‚0.5479411634666017 /loss/FRt2/detection_loss‚1.6171979749790177 0loss/FBt2/detection_loss‚0.45677509264964045 2loss/SRqqt1/assignment_loss‚0.5311129717602118 .loss/FBt1/detection_loss‚1.121089223171434 _runtime‚ 3337.71656991 0loss/FRt1/assignment_loss‚1.8444998278864113 0loss/FRt2/assignment_loss‚0.7520545551410088 1loss/FBt1/assignment_loss‚0.12734882790836022 2loss/FBt2/assignment_loss‚0.048065646870011074 0loss/SRqqt2/detection_loss‚0.701271556686961 trainer/global_step‚3899 % _timestamp‚1.7843608456414433e+09 _step‚320ÀtÿF¼Ú¾‚(:Ï ”ßìÒ‚…‘¼ disk.dm-0.out‚ 11491.859375 network.sent‚ 6.04859e+06 disk.nvme0n1.in‚ 5488.890625! disk.nvme0n1.out‚ 11491.859375 disk.drbd2165.in‚0 cpu‚26.181152127406303( proc.memory.availableMB‚ 604.35546875 network.recv‚907271$ disk./.usageGB‚588.9655113220215 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 5488.890625 disk.sda1.in‚0 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.dm-0.in‚ 5488.890625$ memory_percent‚96.31130695343018" proc.memory.rssMB‚ 2642.7734375 disk.dm-1.out‚ 253.671875 disk.sda2.out‚0* disk./.usagePercent‚39.525969639800266# disk.nvme0c0n1.out‚ 11491.859375 disk.sda3.out‚ 253.671875 disk.drbd2165.out‚0) proc.memory.percent‚16.13020896911621 proc.cpu.threads‚17 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚92.18212890625 disk.sda.out‚ 253.671875 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.in‚0j¤ê“¿‚(:ˆ ”ßìÒ°®©½ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚149.668' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃòÍ‘À‚(:† £ßìÒ‹´ì» gpu.0.gpu‚59.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08OÓçÁ‚(:Ü £ßìÒ‚Âļ disk.rbd1.out‚92.30712890625 disk.rbd0.in‚0 network.sent‚ 6.111234e+06 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 5488.890625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 5488.890625$ disk.nvme0n1.out‚11498.091796875" proc.memory.rssMB‚ 2642.7890625 network.recv‚911726) disk./.usagePercent‚39.52596682371739 disk.sda2.out‚0! disk.dm-0.out‚11498.091796875 disk.rbd1.in‚ 0.01953125! disk.nvme0c0n1.in‚ 5488.890625& disk.nvme0c0n1.out‚11498.091796875 disk.sda.out‚ 254.00390625$ disk./.usageGB‚588.9654693603516 disk.sda1.out‚0 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625) proc.memory.percent‚16.13030433654785 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 cpu‚26.18019570103639' proc.memory.availableMB‚ 604.3359375 disk.sda3.out‚ 254.00390625 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.out‚ 254.00390625$ memory_percent‚96.31142616271973à/˞‚(:“ ²ßìÒ’¹õº gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚238.362' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.672399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0““ÏìÂ(:á ²ßìÒ€€Ø½ disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚11500.97216796875$ memory_percent‚96.31292819976807! proc.memory.rssMB‚ 2642.796875) disk./.usagePercent‚39.52596656770986 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 254.2265625 disk.rbd1.in‚ 0.01953125 network.recv‚915027 disk.drbd2165.out‚0 disk.sda.in‚ 0.00390625* proc.memory.percent‚16.130352020263672 disk.sda2.in‚0 disk.sda.out‚ 254.2265625 disk.dm-0.in‚ 5488.890625 disk.rbd1.out‚92.30712890625 disk.nvme0n1.in‚ 5488.890625 disk.sda3.in‚ 0.00390625$ disk./.usageGB‚588.9654655456543 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 254.2265625( proc.memory.availableMB‚ 604.08984375 proc.cpu.threads‚17 disk.rbd0.out‚0 cpu‚26.179528583723457 network.sent‚ 6.116228e+06# disk.dm-0.out‚11500.97216796875! disk.nvme0c0n1.in‚ 5488.890625& disk.nvme0n1.out‚11500.97216796875üî}2Ä- _wandbruntime‚3377  _runtime‚3377ÿÖ_(ÇÅÁ & lr-AdamW/pg1‚0.0004007477973326067 & lr-AdamW/pg2‚0.0004007477973326067 trainer/global_step‚3949 % _timestamp‚1.7843608857647274e+09 _runtime‚3377.840039983 _step‚321Jä•ñÌÆÆ & lr-AdamW/pg2‚0.0004007477973326067 trainer/global_step‚3949 % _timestamp‚1.7843608857647274e+09 _runtime‚3377.840039983 _step‚321 & lr-AdamW/pg1‚0.0004007477973326067Áˆ™ä×2Ç- _wandbruntime‚3377  _runtime‚3377I‡Éß'ÈÙO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15392614 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6921311 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚603169125331968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36531426 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132310765348864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125331968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4159111183310 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚3949 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15392348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36541403 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595930389598208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4158770633420 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599003990843392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31236254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36531426 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136522116596736 Óšžä'ÉÞO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132310765348864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125331968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4159111183310 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚3949 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15392348 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36541403 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595930389598208 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4158770633420 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599003990843392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31236254 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36531426 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136522116596736 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15392614 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6921311 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚603169125331968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36531426 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ºˆE2Ê- _wandbruntime‚3378  _runtime‚3378­ùBfç ËâM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600236681957222 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31243886 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31243888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15396468 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794357 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3378.456f›Ý581948 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6923056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596076900270080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚3949 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31243886 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8473152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540478 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4166488461824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚603317737374208 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚599151248912384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4166488461824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136556474466816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4212280359424 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784364 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31243888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4166147115520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136556573260288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132344194107392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4160122237144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚603317737374208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15396205 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600237039546584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794357 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6923053 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8473412 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599151231873024 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67784364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1jª¨í&ÌçM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31243888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4166147115520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136556573260288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132344194107392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4160122237144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚603317737374208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15396205 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600237039546584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794357 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6923053 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8473412 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599151231873024 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67784364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600236681957222 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31243886 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31243888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15396468 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794357 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3378.456581948 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6923056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596076900270080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚3949 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31243886 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8473152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540478 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4166488461824 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚603317737374208 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚599151248912384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4166488461824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136556474466816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4212280359424 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784364ÃAÁâ_2Í- _wandbruntime‚3378  _runtime‚3378ž¶'ÕÎÿ /loss/FBt2/detection_loss‚0.5275807490382393 epoch‚9 /loss/FRt1/assignment_loss‚1.904932249795932 2loss/FBt1/assignment_loss‚0.037831915827875186 $loss/total_loss‚9.13670138123552 _runtime‚3378.458676512 _step‚324 /loss/FRt2/assignment_loss‚0.717282879853917 2loss/SRqqt1/assignment_loss‚0.6350546689810934 1loss/SRqqt2/assignment_loss‚0.588203701584791 2loss/FBt2/assignment_loss‚0.010029447586269247 1loss/SRqqt1/detection_loss‚1.0043848940461386 1loss/SRqqt2/detection_loss‚0.6399172313940235 trainer/global_step‚3949 % _timestamp‚1.7843608863836162e+09 /loss/FRt1/detection_loss‚0.5475767703686065 /loss/FRt2/detection_loss‚1.4933185460933736 /loss/FBt1/detection_loss‚1.0305883266652625ÐNéú Ï„ trainer/global_step‚3949 % _timestamp‚1.7843608863836162e+09 /loss/FRt1/detection_loss‚0.5475767703686065 /loss/FRt2/detection_loss‚1.4933185460933736 /loss/FBt1/detection_loss‚1.0305883266652625 /loss/FBt2/detection_loss‚0.5275807490382393 epoch‚9 /loss/FRt1/assignment_loss‚1.904932249795932 2loss/FBt1/assignment_loss‚0.037831915827875186 $loss/total_loss‚9.13670138123552 _runtime‚3378.458676512 _step‚324 /loss/FRt2/assignment_loss‚0.717282879853917 2loss/SRqqt1/assignment_loss‚0.6350546689810934 1loss/SRqqt2/assignment_loss‚0.588203701584791 2loss/FBt2/assignment_loss‚0.010029447586269247 1loss/SRqqt1/detection_loss‚1.0043848940461386 1loss/SRqqt2/detection_loss‚0.6399172313940235ĉëå“Ђ(:ˆ ÁßìÒ«Ýƽ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚244.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ϫ(ïÑ‚(:ä ÁßìÒ›¼“À network.sent‚ 6.121292e+06 disk.rbd1.in‚ 0.01953125 disk.sda.out‚ 254.42578125 disk.dm-0.in‚ 5550.30859375$ disk.nvme0n1.out‚11506.048828125 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625$ memory_percent‚96.31450176239014* proc.memory.percent‚16.130352020263672) disk./.usagePercent‚39.52596298360438 disk.drbd2165.out‚0! disk.dm-0.out‚11506.048828125& disk.nvme0c0n1.out‚11506.048828125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 254.42578125( proc.memory.availableMB‚ 603.83203125$ disk./.usageGB‚588.9654121398926 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 5550.30859375 cpu‚26.178773189989183 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 5550.30859375 disk.dm-1.out‚ 254.42578125 disk.rbd1.out‚92.44775390625 network.recv‚918566 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0! proc.memory.rssMB‚ 2642.796875†½“Ò‚(:ˆ ÐßìÒäÅܺ gpu.0.gpu‚71.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚201.212' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.2424 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0㶂wôÓ‚(:é ÐßìÒ Å༠disk.rbd0.in‚0! disk.nvme0n1.in‚ 5550.30859375 disk.sda.in‚ 0.00390625 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-0.in‚ 5550.30859375 disk.rbd1.out‚92.44775390625# disk./.usageGB‚588.965747833252 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 disk.sda.out‚ 254.87890625 disk.sda3.in‚ 0.00390625! proc.memory.rssMB‚ 2642.796875 network.sent‚ 6.183955e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚11508.29248046875 cpu‚26.178410129097546 network.recv‚922711 disk.sda2.out‚0* proc.memory.percent‚16.130352020263672 disk.sda2.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 5550.30859375 disk.sda3.out‚ 254.87890625$ memory_percent‚96.31330966949463( proc.memory.availableMB‚ 604.02734375) disk./.usagePercent‚39.52598551226739# disk.dm-0.out‚11508.29248046875 disk.sda1.in‚0 disk.dm-1.out‚ 254.87890625& disk.nvme0n1.out‚11508.29248046875´Õ)2Ô- _wandbruntime‚3418  _runtime‚3418ÅZjÇÕÁ & lr-AdamW/pg2‚0.0003998029731017593 trainer/global_step‚3999 % _timestamp‚1.7843609267867932e+09 _runtime‚3418.864227914 _step‚325 & lr-AdamW/pg1‚0.0003998029731017593LjL¢ÌÖÆ & lr-AdamW/pg1‚0.0003998029731017593 & lr-AdamW/pg2‚0.0003998029731017593 trainer/global_step‚3999 % _timestamp‚1.7843609267867932e+09 _runtime‚3418.864227914 _step‚325ÅÀ¢Šc2×- _wandbruntime‚3418  _runtime‚3418ÃwV½à'ØÚO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36985878 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215829990400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133982221887488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138244699190784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4216171329536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31617856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610582230693888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901386 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171329536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613711 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15582982 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8574424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133982203275264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚3999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872315392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚606366059364352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15583236 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872315392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995855 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3418.868256629 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31617854 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606366042324992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36985878 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603255923191808¥¥r"ˆÙßO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138244699190784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4216171329536 TGDeviceStatsMonitor.on† ]%_train_batch_start/allocation.large_pool.allocated‚31617856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610582230693888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901386 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171329536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613711 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15582982 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8574424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133982203275264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚3999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872315392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚606366059364352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15583236 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872315392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995855 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3418.868256629 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31617854 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606366042324992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36985878 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603255923191808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36985878 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215829990400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133982221887488Æ’]¦2Ú- _wandbruntime‚3419  _runtime‚3419Ø‘é&ÛãM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31625488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4217183963648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4263519596544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216842617344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚606513283354624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31625486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4210398174050 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31625486 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610730484357632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630409 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630409 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4210738724052 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31625488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚606513300393984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134015650646016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚606513300393984 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚606513283354624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36994930 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607612832037730 $ _timestamp‚1.784360927406383e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15586705 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7010306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630409 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4210738724052 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31625488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚606513300393984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134015650646016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚606513300393984 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚606513283354624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36994930 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚607612832037730 $ _timestamp‚1.784360927406383e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15586705ÇiœÌ2Ý- _wandbruntime‚3419  _runtime‚3419_ç[7 Þ„ /loss/FBt2/detection_loss‚0.4444428926160475 % _timestamp‚1.7843609274084768e+09 epoch‚9 2loss/SRqqt1/assignment_loss‚0.6508745772296473 3loss/SRqqt2/assignment_loss‚0.48945509988609154 3loss/FBt1/assignment_loss‚0.0028838713328421966 2loss/FBt2/assignment_loss‚0.019228021594019828 1loss/SRqqt2/detection_loss‚0.6243568392426474 _step‚328 %loss/total_loss‚9.188113734101483 trainer/global_step‚3999 /loss/FBt1/detection_loss‚1.2571269318827294 0loss/FRt1/assignment_loss‚1.7945249107161443 .loss/FRt1/detection_loss‚0.536310334988577 1loss/SRqqt1/detection_loss‚1.0043914693238951 _runtime‚3419.483526515 0loss/FRt2/assignment_loss‚0.7583414863154617 /loss/FRt2/detection_loss‚1.6061772989733798Þܓ߉ trainer/global_step‚3999 /loss/FBt1/detection_loss‚1.2571269318827294 0loss/FRt1/assignment_loss‚1.7945249107161443 .loss/FRt1/detection_loss‚0.536310334988577 1loss/SRqqt1/detection_loss‚1.0043914693238951 _runtime‚3419.483526515 0loss/FRt2/assignment_loss‚0.7583414863154617 /loss/FRt2/detection_loss‚1.6061772989733798 /loss/FBt2/detection_loss‚0.4444428926160475 % _timestamp‚1.7843609274084768e+09 epoch‚9 2loss/SRqqt1/assignment_loss‚0.6508745772296473 3loss/SRqqt2/assignment_loss‚0.48945509988609154 3loss/FBt1/assignment_loss‚0.0028838713328421966 2loss/FBt2/assignment_loss‚0.019228021594019828 1loss/SRqqt2/detection_loss‚0.6243568392426474 _step‚328 %loss/total_loss‚9.188113734101483È+]³à‚(:’ ßßìÒ˜ðÙ» gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â™x°óá‚(:è ßßìÒî¢ò¼ network.sent‚ 6.188971e+06! disk.nvme0n1.in‚ 5550.30859375* proc.memory.percent‚16.130352020263672) disk./.usagePercent‚39.52600138473451 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 network.recv‚926321 disk.rbd1.out‚92.51025390625 disk.rbd0.in‚0 disk.dm-0.in‚ 5550.30859375 cpu‚26.1771954340086$ disk./.usageGB‚588.9659843444824( disk.nvme0c0n1.out‚11513.11474609375 disk.sda1.in‚0! proc.memory.rssMB‚ 2642.796875 disk.dm-1.out‚ 255.15234375 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01953125# disk.nvme0c0n1.in‚ 5550.30859375( proc.memory.availableMB‚ 604.03515625# disk.dm-0.out‚11513.11474609375 disk.dm-1.in‚ 0.00390625& disk.nvme0n1.out‚11513.11474609375 disk.sda.in‚ 0.00390625 disk.sda.out‚ 255.15234375 disk.sda3.out‚ 255.15234375 disk.drbd2165.out‚0$ memory_percent‚96.31326198577881Öu)yâ‚(:† îßìÒœ§¶º gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.me½ôÌmoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G.Iïã‚(:ä îßìÒܱѼ disk.sda.in‚ 0.00390625 disk.dm-0.in‚ 5550.30859375# disk.dm-0.out‚11516.24560546875* proc.memory.percent‚16.686081886291504 proc.cpu.threads‚17( disk.nvme0c0n1.out‚11516.24560546875 network.sent‚ 6.251842e+06 network.recv‚930696 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd2165.in‚0$ memory_percent‚96.87068462371826 cpu‚26.1877049498129$ disk./.usageGB‚588.9658393859863 disk.rbd0.out‚0! disk.nvme0n1.in‚ 5550.30859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 5550.30859375 disk.sda.out‚ 255.796875) disk./.usagePercent‚39.52599165644821 disk.sda2.out‚0( proc.memory.availableMB‚ 512.70703125 disk.rbd1.in‚ 0.01953125 disk.drbd2165.out‚0 disk.sda3.out‚ 255.796875 disk.sda2.in‚0& disk.nvme0n1.out‚11516.24560546875 disk.dm-1.out‚ 255.796875 disk.rbd1.out‚92.51025390625# proc.memory.rssMB‚ 2733.84765625ÔP2ä- _wandbruntime‚3438  _runtime‚3438}Ú‘³Ê=åÄ{ >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 'particle/accuracy‚0.853135883808136 'particle/f_score‚0.6460714340209961 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3112572729587555 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893744587898254 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7674975395202637 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8397958874702454 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6905847787857056 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6555527448654175 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7362058758735657 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6validation_average_jet_accuracy‚0.3833577334880829 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4114583432674408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7076333165168762 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21338389813899994 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6592504978179932 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 jet/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚329 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9792395234107971 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6865941286087036 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25122079253196716 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15426599979400635 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2127462774515152 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6369631886482239 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6442810893058777 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_2_of_3‚0.5810576677322388 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4997762441635132 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843609460304017e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30179524421691895 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04260490462183952 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8301966786384583 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7510493993759155 jet/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3172324001789093 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚8.486364643049713 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896196722984314 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2625032961368561 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6242303252220154 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47235652804374695 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7538387179374695 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8796244263648987 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7367486357688904 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_3_of_3‚0.07462942600250244 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24376413226127625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6335034370422363 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04391787573695183 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39111894369125366 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0486011803150177 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04007458686828613 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03767029941082001 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997212827205658 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5911471247673035 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5687500238418579 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6550586819648743 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7610922455787659 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09259232133626938 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21136724948883057 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_3‚0.9270519018173218 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959481358528137 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9002807140350342 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6436338424682617 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25122079253196716 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5645017027854919 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4631898105144501 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9664354920387268 *jet/accuracy_2_of_4‚0.8057062029838562 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_5_of_5‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 _runtime‚3438.106579126 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968703985214233 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05078630894422531 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.941774845123291 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.387875497341156 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1381443291902542 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6555527448654175 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0390421561896801 +jet/accuracy_2_of_2‚0.15921175479888916 /particle/accuracy_4_of_4‚0.9132212400436401 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7378413081169128 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.660990834236145 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4794990122318268 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8397958874702454 jet/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22522111237049103 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2118275910615921 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.987639844417572 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12586770951747894 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26027151942253113 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25048935413360596 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6417302489280701 +jet/accuracy_3_of_4‚0.09061292558908463 trainer/global_step‚3999 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47016817331314087 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6505110263824463 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07596281170845032 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950067400932312 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.198985755443573 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7942591905593872 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3427564799785614 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19076582789421082 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9157426953315735 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6904249787330627 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8713868260383606 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7902674674987793 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7477920055389404 particle/accuracy_2_of_2‚1 epoch‚9 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8366591930389404 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22988076508045197 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46596989035606384 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8223356604576111 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.775357723236084 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8421769142150879 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2627577781677246 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.791277289390564 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901041865348816 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.732143759727478 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03767029941082001 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6993592381477356 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.679732620716095 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7421048283576965 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7621231079101562 particle/accuracy_1_of_1‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_2_of_4‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7009918689727783 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850085973739624 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6701678037643433 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.044369425624608994 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6812185645103455 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9215726256370544 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 /particle/accuracy_3_of_4‚0.9910430908203125 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.042916953563690186 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5653331875801086 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9848009943962097 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939244389533997 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20122206211090088 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21670137345790863 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3377140462398529 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940598011016846 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937974810600281 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958851337432861 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2087859809398651 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2627890408039093 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8426116108894348 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17749999463558197 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2243041694164276 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9927998781204224 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6799973845481873 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7429577112197876 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1414032131433487 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23982146382331848 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7510319948196411 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5|ý†Ú<æÉ{ +jet/accuracy_3_of_3‚0.07462942600250244 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24376413226127625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6335034370422363 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04391787573695183 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39111894369125366 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0486011803150177 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04007458686828613 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03767029941082001 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997212827205658 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5911471247673035 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5687500238418579 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6550586819648743 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7610922455787659 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09259232133626938 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21136724948883057 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_3‚0.9270519018173218 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959481358528137 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9002807140350342 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6436338424682617 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25122079253196716 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5645017027854919 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4631898105144501 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9664354920387268 *jet/accuracy_2_of_4‚0.8057062029838562 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_5_of_5‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 _runtime‚3438.106579126 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968703985214233 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05078630894422531 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.941774845123291 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.387875497341156 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1381443291902542 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6555527448654175 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0390421561896801 +jet/accuracy_2_of_2‚0.15921175479888916 /particle/accuracy_4_of_4‚0.9132212400436401 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7378413081169128 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.660990834236145 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4794990122318268 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8397958874702454 jet/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22522111237049103 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2118275910615921 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.987639844417572 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12586770951747894 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26027151942253113 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25048935413360596 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6417302489280701 +jet/accuracy_3_of_4‚0.09061292558908463 trainer/global_step‚3999 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47016817331314087 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6505110263824463 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07596281170845032 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950067400932312 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.198985755443573 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7942591905593872 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3427564799785614 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19076582789421082 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9157426953315735 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6904249787330627 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8713868260383606 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7902674674987793 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7477920055389404 particle/accuracy_2_of_2‚1 epoch‚9 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8366591930389404 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22988076508045197 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46596989035606384 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8223356604576111 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.775357723236084 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8421769142150879 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2627577781677246 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.791277289390564 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901041865348816 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.732143759727478 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03767029941082001 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6993592381477356 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.679732620716095 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7421048283576965 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7621231079101562 particle/accuracy_1_of_1‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_2_of_4‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7009918689727783 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850085973739624 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6701678037643433 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.044369425624608994 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6812185645103455 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9215726256370544 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 /particle/accuracy_3_of_4‚0.9910430908203125 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.042916953563690186 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5653331875801086 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9848009943962097 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939244389533997 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20122206211090088 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21670137345790863 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3377140462398529 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940598011016846 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937974810600281 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958851337432861 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2087859809398651 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2627890408039093 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8426116108894348 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17749999463558197 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2243041694164276 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9927998781204224 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6799973845481873 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7429577112197876 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1414032131433487 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23982146382331848 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7510319948196411 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 'particle/accuracy‚0.853135883808136 'particle/f_score‚0.6460714340209961 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3112572729587555 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893744587898254 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7674975395202637 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8397958874702454 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6905847787857056 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6555527448654175 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7362058758735657 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6validation_average_jet_accuracy‚0.3833577334880829 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4114583432674408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7076333165168762 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21338389813899994 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6592504978179932 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 jet/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚329 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9792395234107971 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6865941286087036 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25122079253196716 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15426599979400635 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2127462774515152 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6369631886482239 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6442810893058777 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_2_of_3‚0.5810576677322388 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4997762441635132 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843609460304017e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30179524421691895 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04260490462183952 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8301966786384583 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7510493993759155 jet/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3172324001789093 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚8.486364643049713 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896196722984314 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2625032961368561 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6242303252220154 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47235652804374695 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.01278831809"‰Eõ75914 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7538387179374695 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8796244263648987 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7367486357688904 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875É®ê~0“ç‚(:ˆ ýßìÒúªÊ¼ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÁç†ìè‚(:á ýßìÒ˜Úæ¼# disk.dm-0.out‚12044.99462890625 disk.sda1.in‚0( proc.memory.availableMB‚ 829.76953125 cpu‚26.18789944489748 disk.sda.out‚ 256.1953125 proc.memory.rssMB‚ 2737.65625 network.sent‚ 6.256834e+06 network.recv‚934209 disk.dm-0.in‚ 5626.7109375 disk.dm-1.out‚ 256.1953125 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚92.57275390625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 5626.7109375 disk.sda2.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚12044.99462890625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 256.1953125 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625( disk.nvme0c0n1.out‚12044.99462890625 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚588.9658088684082 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 5626.7109375# memory_percent‚94.9354887008667* proc.memory.percent‚16.709327697753906) disk./.usagePercent‚39.52598960838793 disk.drbd2165.out‚0Œ'‘é‚(:† ŒàìÒš®¸º gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÁÔîê‚(:ã ŒàìÒ¶Ÿù½ disk.nvme0n1.in‚ 5626.7109375 disk.sda.in‚ 0.00390625 disk.sda2.in‚0 disk.dm-1.out‚ 256.53515625* proc.memory.percent‚16.709327697753906 disk.dm-1.in‚ 0.00390625 network.sent‚ 6.314536e+06 network.recv‚938488$ disk./.usageGB‚588.9658889770508 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚92.57275390625 proc.memory.rssMB‚ 2737.65625 proc.cpu.threads‚17) disk./.usagePercent‚39.52599498454616( disk.nvme0c0n1.out‚12052.94189453125& disk.nvme0n1.out‚12052.94189453125 disk.sda3.out‚ 256.53515625 disk.sda2.out‚0 disk.drbd2165.in‚0 cpu‚26.18728880030949 disk.sda.out‚ 256.53515625 disk.sda1.out‚0# disk.dm-0.out‚12052.94189453125& proc.memory.availableMB‚ 496.984375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 5626.7109375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 5626.7109375 disk.sda3.in‚ 0.00390625$ memory_percent‚96.96664810180664€Í+‘ë‚(:† ›àìÒÞ¹ï» gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚246.444' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èRUÛïì‚(:ä ›àìÒùÍô¼) disk./.usagePercent‚39.52599344850095& disk.nvme0n1.out‚12058.08544921875 disk.sda2.out‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2737.65625 disk.nvme0n1.in‚ 5626.7109375 disk.dm-0.in‚ 5626.7109375 disk.rbd1.out‚92.65087890625 cpu‚26.186503074199216 disk.drbd2165.in‚0 disk.dm-1.out‚ 256.87109375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12058.08544921875# memory_percent‚96.9667911529541' proc.memory.availableMB‚ 496.9609375* proc.memory.percent‚16.709327697753906$ disk./.usageGB‚588.9658660888672 disk.sda3.out‚ 256.87109375# disk.dm-0.out‚12058.08544921875 disk.rbd1.in‚ 0.01953125 proc.cpu.threads‚17 disk.sda.out‚ 256.87109375 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 network.sent‚ 6.319505e+06 network.recv‚941897 disk.sda1.out‚0 disk.sda.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625" disk.nvme0c0n1.in‚ 5626.7109375^š÷2í- _wandbruntime‚3480  _runtime‚3480ÛGÇîÁ _runtime‚3480.553228213 _step‚330 & lr-AdamW/pg1‚0.0003988464398844182 & lr-AdamW/pg2‚0.0003988464398844182 trainer/global_step‚4049 % _timestamp‚1.7843609884758148e+09fG±TÌïÆ % _timestamp‚1.7843609884758148e+09 _runtime‚3480.553228213 _step‚330 & lr-AdamW/pg1‚0.0003988464398844182 & lr-AdamW/pg2‚0.0003988464398844182 trainer/global_step‚4049Ê¡²k2ð- _wandbruntime‚3480  _runtime‚3480üo›ß'ñÙO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619942277535744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615657660456960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15809536 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32120122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32120122 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135669571913728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4331892225536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32120120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612490137038848 trainer/global_step‚4049 _step‚331 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4284258700288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8710938 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140001445527040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32120120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8711205 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _runtime‚3480.557235959 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630777 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616767843331958 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4277723332470 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135669553301504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7098328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919157248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4284600039424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919157248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615657677496320 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4278063882360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4284258700288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4284600039424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37520634 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144Zu…ä'òÞO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619942277535744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37520634 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15809266 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615657660456960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140001540133376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37520634 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640756 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619942277535744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615657660456960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15809536 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32120122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32120122 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135669571913728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4331892225536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32120120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612490137038848 trainer/global_step‚4049 _step‚331 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4284258700288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8710938 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140001445527040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32120120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8711205 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _runtime‚3480.557235959 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630777 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616767843331958 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4277723332470 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135669553301504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7098328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919157248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4284600039424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919157248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚615657677496320 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4278063882360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4284258700288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4284600039424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657454 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32127754 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135702982060032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271327232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4285271327232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚3481.170566781 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7100076 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4278734386192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612636647710720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7100073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784360989095024e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69657454 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8713017 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279074936194 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15813090 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647461 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615804918525952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140035997114368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135703000672256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4332996442112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140035889932288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4285612673536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531199488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269cÍ-î&õèM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135703000672256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4332996442112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140035889932288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4285612673536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531199488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657454 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32127754 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135702982060032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271327232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4285271327232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚3481.170566781 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7100076 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4278734386192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612636647710720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7100073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784360989095024e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69657454 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8713017 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279074936194 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15813090 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647461 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615804918525952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140035997114368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888Ì™WVì2ö- _wandbruntime‚3481  _runtime‚3481ÿ<\ ÷„ /loss/FBt2/detection_loss‚0.4295177001892454 _runtime‚3481.172265098 1loss/SRqqt2/detection_loss‚0.6259464549687088 %loss/total_loss‚8.204379373074968 % _timestamp‚1.7843609890972044e+09 0loss/FRt2/assignment_loss‚0.7866561822429102 2loss/SRqqt1/assignment_loss‚0.5298833209878606 2loss/FBt1/assignment_loss‚0.003820249787278015 .loss/FRt1/detection_loss‚0.543566990333789 epoch‚10 _step‚333 0loss/FRt1/assignment_loss‚1.7600211797473586 3loss/SRqqt2/assignment_loss‚0.49686601011111153 /loss/FRt2/detection_loss‚1.4624248124773938 /loss/FBt1/detection_loss‚0.6156108728459766 trainer/global_step‚4049 2loss/FBt2/assignment_loss‚0.010639041444050939 1loss/SRqqt1/detection_loss‚0.9394265579392821 Éßø‰ 0loss/FRt2/assignment_loss‚0.7866561822429102 2loss/SRqqt1/assignment_loss‚0.5298833209878606 2loss/FBt1/assignment_loss‚0.003820249787278015 .loss/FRt1/detection_loss‚0.543566990333789 epoch‚10 _step‚333 0loss/FRt1/assignment_loss‚1.7600211797473586 3loss/SRqqt2/assignment_loss‚0.49686601011111153 /loss/FRt2/detection_loss‚1.4624248124773938 /loss/FBt1/detection_loss‚0.6156108728459766 trainer/global_step‚4049 2loss/FBt2/assignment_loss‚0.010639041444050939 1loss/SRqqt1/detection_loss‚0.9394265579392821 /loss/FBt2/detection_loss‚0.4295177001892454 _runtime‚3481.172265098 1loss/SRqqt2/detection_loss‚0.6259464549687088 %loss/total_loss‚8.204379373074968 % _timestamp‚1.7843609890972044e+09Í–Ê[Lù‚(:’ ªàìÒå“á¼ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q¥ß*ãú‚(:Ø ªàìÒÚÜè½& proc.memory.availableMB‚ 497.171875 network.sent‚ 6.324576e+06$ disk./.usageGB‚588.9657745361328 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚257.1875 disk.sda1.in‚0$ memory_percent‚96.96550369262695 disk.rbd0.in‚0) disk./.usagePercent‚39.52598730432013 disk.rbd1.in‚ 0.01953125( disk.nvme0c0n1.out‚12061.08740234375& disk.nvme0n1.out‚12061.08740234375 disk.sda.out‚257.1875 cpu‚26.185907684135667 disk.dm-0.in‚ 5626.7109375 disk.dm-1.out‚257.1875 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625* proc.memory.percent‚16.709327697753906 disk.rbd1.out‚92.65087890625 disk.rbd0.out‚0 proc.memory.rssMB‚ 2737.65625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 5626.7109375 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚945193# disk.dm-0.out‚12061.08740234375" disk.nvme0c0n1.in‚ 5626.7109375Ó£}[×û‚(:Ì ¹àìÒ¢“ݼ* disk./.usagePercent‚39.525988328350266 proc.memory.rssMB‚ 2737.65625 proc.cpu.threads‚17 disk.sda1.in‚0( proc.memory.availableMB‚ 496.98828125 cpu‚26.185064431422038 disk.dm-1.out‚257.6875 disk.sda2.in‚0 disk.rbd1.in‚ 0.01953125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.out‚257.6875* proc.memory.percent‚16.709327697753906 network.sent‚ 6.387315e+06$ disk./.usageGB‚588.9657897949219 disk.drbd2165.out‚0 disk.sda3.in‚ 0.00390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚5699.375 disk.nvme0n1.in‚5699.375% disk.nvme0n1.out‚12065.8017578125 disk.sda.out‚257.6875$ memory_percent‚96.96662425994873 disk.sda2.out‚0" disk.dm-0.out‚12065.8017578125 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚92.71337890625 network.recv‚949529 disk.sda1.out‚0 disk.dm-0.in‚5699.375' disk.nvme0c0n1.out‚12065.8017578125 disk.sda.in‚ 0.00390625wŽ|“ü‚(:ˆ ¹àìÒ™žõ¼ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…£*2ý- _wandbruntime‚3521  _runtime‚3521ö¤IrÇþÁ & lr-AdamW/pg2‚0.0003978782591169243 trainer/global_step‚4099 % _timestamp‚1.7843610294537737e+09 _runtime‚3521.529119126 _step‚334 & lr-AdamW/pg1‚0.0003978782591169243ìæö¾ÌÿÆ & lr-AdamW/pg2‚0.0003978782591169243 trainer/global_step‚4099 % _timestamp‚1.7843610294537737e+09 _runtime‚3521.529119126 _step‚334 & lr-AdamW/pg1‚0.0003978782591169243ÎiÂ2€- _wandbruntime‚3521  _runtime‚3521¦y·@à'ÚO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37965109 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627354666140672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚4099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137340991227904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7185581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7185578 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666140672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚623019711938560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954202112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627355024519168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624143993412466 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137341009840128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚335 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37965109 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4383189431808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4383278008832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚623019728977920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4334954202112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32501722 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8814412 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619815653593088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16000248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8814667 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269m’ÜCì‚ßO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4328339819378 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843610294545822e+09 _runtime‚3521.530043431 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32501722 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8814412 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚619815653593088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16000248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8814667 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37965109 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627354666140672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚4099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137340991227904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7185581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7185578 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666140672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚623019711938560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954202112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627355024519168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624143993412466 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137341009840128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚335 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37965109 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4383189431808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4383278008832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚623019728977920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4334954202112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623166952968192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37984152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137374438598656 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493506 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843610300704784e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141758638469120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627503278182912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493506 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37984152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32509354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4336308175360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32509352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16003996 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141758741456896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32509354 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16003731 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8816670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8816408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4384302858240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37974161 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619962181304320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976¸öÏ=î&…èM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141758741456896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32509354 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16003731 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8816670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8816408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4384302858240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37974161 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619962181304320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚623166952968192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137374419986432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619962164264960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70483513 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7187326 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7187323 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32509352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627502919797248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚624291872727422 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 3522.14608172 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 trainer/global_step‚4099 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624291515138060 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627503278182912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚623166970007552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623166952968192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37984152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137374438598656 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493506 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7843610300704784e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141758638469120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627503278182912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493506 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37984152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32509354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4336308175360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32509352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16003996Й›a2†- _wandbruntime‚3522  _runtime‚3522r4u:‡ 2loss/FBt1/assignment_loss‚0.004368752197675663 2loss/FBt2/assignment_loss‚0.005690246468452675 /loss/FRt2/detection_loss‚1.4914837927411395 0loss/SRqqt1/detection_loss‚0.918811947908316 /loss/FBt1/detection_loss‚0.8783411442851409 /loss/FBt2/detection_loss‚0.5055409508277627 /loss/FRt1/assignment_loss‚1.879928040606247 0loss/FRt2/assignment_loss‚0.7551320238470752 epoch‚10 _step‚337 2loss/SRqqt1/assignment_loss‚0.6804695311860152 2loss/SRqqt2/assignment_loss‚0.5644234476423073 %loss/total_loss‚8.867024887003387 trainer/global_step‚4099 _runtime‚3522.147618251 /loss/FRt1/detection_loss‚0.5267358539801951 1loss/SRqqt2/detection_loss‚0.6560991553130598 $ _timestamp‚1.784361030072553e+09Œn™¢ ˆ† /loss/FRt1/detection_loss‚0.5267358539801951 1loss/SRqqt2/detection_loss‚0.6560991553130598 $ _timestamp‚1.784361030072553e+09 2loss/FBt1/assignment_loss‚0.004368752197675663 2loss/FBt2/assignment_loss‚0.005690246468452675 /loss/FRt2/detection_loss‚1.4914837927411395 0loss/SRqqt1/detection_loss‚0.918811947908316 /loss/FBt1/detection_loss‚0.8783411442851409 /loss/FBt2/detection_loss‚0.5055409508277627 /loss/FRt1/assignment_loss‚1.879928040606247 0loss/FRt2/assignment_loss‚0.7551320238470752 epoch‚10 _step‚337 2loss/SRqqt1/assignment_loss‚0.6804695311860152 2loss/SRqqt2/assignment_loss‚0.5644234476423073 %loss/total_loss‚8.867024887003387 trainer/global_step‚4099 _runtime‚3522.147618251Ñ®Î݉‚(:’ ÈàìÒš›é» gpu.0.gpu‚58.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚211.706' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚42.34119999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ǺI㊂(:Ø ÈàìÒô´‘½ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚5699.375 disk.nvme0c0n1.in‚5699.375 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.out‚ 258.04296875 disk.nvme0n1.in‚5699.375 network.sent‚ 6.392287e+06 network.recv‚952919 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0 disk.sda1.out‚0" disk.dm-0.out‚12068.3623046875' disk.nvme0c0n1.out‚12068.3623046875 disk.sda.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625$ memory_percent‚96.97229862213135 proc.memory.rssMB‚ 2737.65625% disk.nvme0n1.out‚12068.3623046875* disk./.usagePercent‚39.525988840365336 disk.sda1.in‚0 disk.sda.out‚ 258.04296875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 258.04296875( proc.memory.availableMB‚ 496.05859375 cpu‚26.184416997074283$ disk./.usageGB‚588.9657974243164 disk.rbd1.out‚92.71337890625* proc.memory.percent‚16.709327697753906¨^“‹‚(:ˆ ×àìÒ“º˜» gpu.0.gpu‚84.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 y/ÜŒ‚(:Ñ ×àìÒÚ¦ì½ disk.rbd0.in‚0 proc.memory.rssMB‚ 2737.65625* proc.memory.percent‚16.709327697753906 cpu‚26.183840316132233 network.recv‚957321$ disk./.usageGB‚588.9658050537109 disk.sda3.out‚ 258.734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚5699.375 disk.sda.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 disk.rbd0.out‚0 disk.nvme0n1.in‚5699.375 disk.sda2.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚12075.07080078125 proc.cpu.threads‚17 network.sent‚ 6.454955e+06& disk.nvme0n1.out‚12075.07080078125# memory_percent‚96.9733715057373( disk./.usagePercent‚39.5259893523804 disk.rbd1.out‚92.85400390625' proc.memory.availableMB‚ 495.8828125( disk.nvme0c0n1.out‚12075.07080078125 disk.sda1.in‚0 disk.dm-0.in‚5699.375 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 258.734375 disk.sda3.in‚ 0.00390625 disk.dm-1.out‚ 258.734375Q!‹Û‚(:Ð æàìÒïß°¼ cpu‚26.183092332689867 network.sent‚ 6.459159e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 259.2421875$ disk.nvme0n1.out‚12079.337890625$ memory_percent‚96.97051048278809 proc.cpu.threads‚17 disk.sda3.out‚ 259.2421875 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚5699.375! disk.dm-0.out‚12079.337890625 disk.rbd1.out‚92.85400390625* proc.memory.percent‚16.709327697753906 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚12079.337890625 disk.nvme0n1.in‚5699.375 disk.sda.in‚ 0.00390625 network.recv‚959854) disk./.usagePercent‚39.52597168786054 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 259.2421875 proc.memory.rssMB‚ 2737.65625$ disk./.usageGB‚588.9655418395996 disk.sda1.out‚0 disk.dm-0.in‚5699.375 disk.rbd1.in‚ 0.01953125 disk.sda3.in‚ 0.00390625' proc.memory.availableMB‚ 496.3515625O|ãM“Ž‚(:ˆ æàìÒ‚ŠŽ½ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+%92- _wandbruntime‚3562  _runtime‚3562P§ºÜÉà ' lr-AdamW/pg1‚0.00039689849298371924 ' lr-AdamW/pg2‚0.00039689849298371924 trainer/global_step‚4149 % _timestamp‚1.7843610702727356e+09 _runtime‚3562.348180377 _step‚338çå/™Î‘È ' lr-AdamW/pg1‚0.00039689849298371924 ' lr-AdamW/pg2‚0.00039689849298371924 trainer/global_step‚4149 % _timestamp‚1.7843610702727356e+09 _runtime‚3562.348180377 _step‚338ÒÔ0Öß2’- _wandbruntime‚3562  _runtime‚3562Ø4€ã“ÚO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4378956306286 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚3562.349356868 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771502592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991043072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429538 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843610702737691e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627141204226048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991043072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429538 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143446821691904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4434392537600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492974 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚339 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630381780459520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143446935172608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312860 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8918489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302881 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413124096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630381763420160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7272828 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4149 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚634767413124096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8918232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649703936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16191060 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4434487406080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚631520501082224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888à˜,oå'”ßO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7272828 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4149 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚634767413124096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8918232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649703936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16191060 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4434487406080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚631520501082224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4378956306286 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚3562.349356868 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771502592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991043072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429538 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843610702737691e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627141204226048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991043072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429538 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143446821691904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4434392537600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492974 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚339 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419561 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630381780459520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143446935172608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312860 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8918489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302881 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413124096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630381763420160Ó/ñ¼Å2•- _wandbruntime‚3562  _runtime‚3562ÝÌĉé&–ãM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚634916025166336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38428613 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚634915666780672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32890952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025166336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627287697858560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319565 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329558 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16194852 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379967360008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 _step‚340 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630529004449792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 $ _timestamp‚1.784361070888339e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380307910010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386662330880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8920538 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4387003677184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319565 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139045876525056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329558 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16194852 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379967360008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 _step‚340 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630529004449792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 $ _timestamp‚1.784361070888339e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380307910010 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386662330880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8920538 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4387003677184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319565 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139045876525056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚634915666780672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32890952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025166336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2Ô­ð»2˜- _wandbruntime‚3562  _runtime‚3562DÑ™€ /loss/FBt1/detection_loss‚0.6986630722372369 epoch‚10 0loss/FRt1/assignment_loss‚1.9041382480091817 1loss/SRqqt2/assignment_loss‚0.524568200104502 1loss/FBt2/assignment_loss‚0.04981806305987235 /loss/FBt2/detection_loss‚0.5558709333554896 trainer/global_step‚4149 % _timestamp‚1.7843610708899522e+09 2loss/SRqqt1/assignment_loss‚0.7184494930549116 3loss/FBt1/assignment_loss‚0.0017002363611440655 ,loss/FRt2/detection_loss‚1.5118480514454 1loss/SRqqt1/detection_loss‚0.9686249748665945 _step‚341 0loss/FRt2/assignment_loss‚0.7647102362048017 1loss/SRqqt2/detection_loss‚0.6625327673053216 %loss/total_loss‚8.852080421378975 _runtime‚3562.965038542 /loss/FRt1/detection_loss‚0.4911561453745174®DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚642180160107520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33264920 % _timestamp‚1.7843611112290761e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160107520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145169457894912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38883990 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637743814901760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436686544896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16381491 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9021413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4485471042048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637743814901760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16381771 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4429572793194 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638896651162732 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33264920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637743831941120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4429913343084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33264922 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436686544896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚642180518486016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚140683885692928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138933 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436345205760 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9021690 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4485572201984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4485572201984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚642180160107520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33264920 % _timestamp‚1.7843611112290761e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160107520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145169457894912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38883990 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637743814901760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436686544896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16381491 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9021413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4485471042048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637743814901760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16381771 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4429572793194 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638896651162732 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33264920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637743831941120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4429913343084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33264922 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436686544896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚642180518486016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚140683885692928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138933 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436345205760 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9021690 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚3603.916858641 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚637891072970752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784361111841242e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 _step‚344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165610 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4430924396918 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165610 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9023429 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16385252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699179008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642328772149760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145203785454592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72155617 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9023696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145203898928128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚140717314451456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33272554 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155617‘)Àýî&©èM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33272552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚637891055931392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437357832704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚637891072970752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784361111841242e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 _step‚344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165610 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4430924396918 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165610 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9023429 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16385252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699179008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642328772149760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145203785454592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72155617 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9023696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145203898928128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚140717314451456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33272554 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155617 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38893056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639043815299076 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚634613231452160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38893056 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38883065 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚3603.916858641 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907090944 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚641792271413248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚645105866383360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487040707584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142355323619328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646272443653990 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4480529829992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487382046720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4536801522688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649593265469440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646272801243240 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚347 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment¼å—+´ .small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645105866383360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚641792254373888 trainer/global_step‚4249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33646522 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142355305007104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72974985 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39338442 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39338442 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚645105883422720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9124907 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4536708751360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9124636 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649592907090944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39328465 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382046720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33646520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39328465 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843611522445226e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508€¿ïeå'¶ßO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚645105866383360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487040707584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142355323619328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646272443653990 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4480529829992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487382046720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4536801522688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649593265469440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646272801243240 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚347 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645105866383360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚641792254373888 trainer/global_step‚4249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33646522 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142355305007104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72974985 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39338442 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39338442 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚645105883422720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9124907 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4536708751360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9124636 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649592907090944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39328465 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382046720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33646520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39328465 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843611522445226e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907090944 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚641792271413248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888Ûõ_ÿj2·- _wandbruntime‚3644  _runtime‚3644(Ãé&¸ãM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146926461307392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843611528567004e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33654152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33654152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337517 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645253107412992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16575800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968946 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚348 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39337517 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9126727 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4537818208768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053334528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33654154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚649741160747520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645253124452352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001662 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394680832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _runtime‚ 3644.93226779 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649741519133184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488394680832 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚649741519133184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7449076 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053334528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142388733765632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888""“4)$¹èM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚649741519133184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7449076 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053334528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142388733765632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146926461307392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843611528567004e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33654152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33654152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337517 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645253107412992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16575800 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968946 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚348 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39337517 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9126727 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4537818208768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053334528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33654154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚649741160747520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645253124452352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001662 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394680832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _runtime‚ 3644.93226779 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649741519133184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488394680832 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_trainwä (Å_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PöÁ‚(:ë ÞáìÒœøš¼ cpu‚26.17864245605059 network.sent‚ 6.678001e+06 disk.sda3.out‚ 261.9609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5829.47265625 proc.memory.rssMB‚ 2737.65625 disk.sda3.in‚ 0.00390625# disk.dm-0.out‚12110.99755859375 disk.rbd0.out‚0! disk.nvme0n1.in‚ 5829.47265625 proc.cpu.threads‚17 network.recv‚ 1.004069e+06) disk./.usagePercent‚39.52594199098657 disk.sda2.out‚0 disk.dm-1.out‚ 261.9609375 disk.sda.out‚ 261.9609375 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625& disk.nvme0n1.out‚12110.99755859375$ memory_percent‚96.96881771087646* proc.memory.percent‚16.709327697753906$ disk./.usageGB‚588.9650993347168 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚93.13525390625 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 5829.47265625( disk.nvme0c0n1.out‚12110.99755859375 disk.sda.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125( proc.memory.availableMB‚ 496.62890625LG^“‚(:ˆ ÞáìÒñ¾Ã¼ gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚244.554' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¹Î2Ã- _wandbruntime‚3685  _runtime‚3685üY1„ÉÄà % _timestamp‚1.7843611930167875e+09 _runtime‚3685.092209792 _step‚350 ' lr-AdamW/pg1‚0.00039389031537154966 ' lr-AdamW/pg2‚0.00039389031537154966 trainer/global_step‚4299;o¶äÎÅÈ ' lr-AdamW/pg1‚0.00039389031537154966 ' lr-AdamW/pg2‚0.00039389031537154966 trainer/global_step‚4299 % _timestamp‚1.7843611930167875e+09 _runtime‚3685.092209792 _step‚350Þ ¤2Æ- _wandbruntime‚3685  _runtime‚3685lã6ß'ÇÙO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34028120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649117805006848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9227449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657005654074368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16762030 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34028120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148614561536512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4531146316900 _step‚351 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811037 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652467934904320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654074368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077548544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16761766 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7534581 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657006012452864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148614668725760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚653648951323748 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649117787967488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 $ _timestamp‚1.784361193020541e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652467917864960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚4299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652467934904320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39792894 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077548544 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537736209408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4587818603008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9227188 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39782917 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34028122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4587907180032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537736209408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚653648593734498 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34028122 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4530805767010 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077548544 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537736209408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4587818603008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9227188 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39782917 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34028122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4587907180032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537736209408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚653648593734498 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34028122 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4530805767010 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654074368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077548544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578ßãŒ)ª2É- _wandbruntime‚3685  _runtime‚3685µ—¦gòÊáM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657154266116608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16765549 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9229226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4539090182656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649264298639360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4588853314560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4532157370734 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39791969 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657154266116608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652615158894592 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚657153907730944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9229486 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652615175933952 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34035752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.onò%„õ_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4531816820732 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚4299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39801960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653796115460092 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚649264315678720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538748836352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73837714 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837714 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39801960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049454 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34035752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148649025006592 _step‚352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827721 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657154266116608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16765549 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9229226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4539090182656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649264298639360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4588853314560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4532157370734 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39791969 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657154266116608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652615158894592 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚657153907730944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9229486 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652615175933952 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34035752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4531816820732 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚4299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39801960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653796115460092 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚649264315678720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4538748836352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73837714 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837714 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39801960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049454 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34035752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148649025006592 _step‚352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827721 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616à‚u2Ì- _wandbruntime‚3685  _runtime‚3685ð ØÍ _runtime‚3685.710250127 _step‚353 1loss/FBt2/assignment_loss‚0.02739970211169316 0loss/FRt1/assignment_loss‚1.7179118070189514 2loss/SRqqt2/assignment_loss‚0.5198929584333861 /loss/FRt2/detection_loss‚1.5536312782159762 1loss/SRqqt1/detection_loss‚0.9157161484549806 1loss/SRqqt2/detection_loss‚0.6435440090096235 /loss/FBt1/detection_loss‚0.7817825831532959 /loss/FBt2/detection_loss‚0.5478591032735863 %loss/total_loss‚8.605731416088677 0loss/FRt2/assignment_loss‚0.7998493554556898 0loss/SRqqt1/assignment_loss‚0.55825413718388 2loss/FBt1/assignment_loss‚0.012292881587818043 /loss/FRt1/detection_loss‚0.5275974521897954 epoch‚10 trainer/global_step‚4299 % _timestamp‚1.7843611936352067e+09œf{ì Ά 0loss/FRt1/assignment_loss‚1.7179118070189514 2loss/SRqqt2/assignment_loss‚0.5198929584333861 /loss/FRt2/detection_loss‚1.5536312782159762 1loss/SRqqt1/detection_loss‚0.9157161484549806 1loss/SRqqt2/detection_loss‚0.6435440090096235 /loss/FBt1/detection_loss‚0.7817825831532959 /loss/FBt2/detection_loss‚0.5478591032735863 %loss/total_loss‚8.605731416088677 0loss/FRt2/assignment_loss‚0.7998493554556898 0loss/SRqqt1/assignment_loss‚0.55825413718388 2loss/FBt1/assignment_loss‚0.012292881587818043 /loss/FRt1/detection_loss‚0.5275974521897954 epoch‚10 trainer/global_step‚4299 % _timestamp‚1.7843611936352067e+09 _runtime‚3685.710250127 _step‚353 1loss/FBt2/assignment_loss‚0.02739970211169316áYÓ®“Ï‚(:ˆ íáìÒ݈”» gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E…ѵöЂ(:ë íáìÒæÛ¼# disk./.usageGB‚588.965145111084 disk.drbd2165.out‚0 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 496.3515625# disk.nvme0c0n1.in‚ 5889.37890625$ memory_percent‚96.97051048278809' disk.nvme0c0n1.out‚12116.7978515625! disk.nvme0n1.in‚ 5889.37890625 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚93.19775390625 disk.rbd0.out‚0 disk.sda3.out‚ 262.18359375 disk.sda1.out‚0* disk./.usagePercent‚39.525945063076975 disk.dm-1.out‚ 262.18359375 disk.sda.in‚ 0.00390625 proc.memory.rssMB‚ 2737.65625 network.sent‚ 6.681238e+06 network.recv‚ 1.007622e+06 disk.dm-0.in‚ 5889.37890625% disk.nvme0n1.out‚12116.7978515625 disk.sda.out‚ 262.18359375* proc.memory.percent‚16.709327697753906 disk.drbd2165.in‚0" disk.dm-0.out‚12116.7978515625 disk.rbd0.in‚0 cpu‚26.177644198363794 proc.cpu.threads‚17 disk.sda1.in‚0žë¸,‘Ñ‚(:† üáìÒ«µ«» gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J[øÒ‚(:í üáìÒ§ð„½ disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 5889.37890625( disk.nvme0c0n1.out‚12119.96142578125! disk.nvme0n1.in‚ 5889.37890625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 262.50390625 cpu‚26.177260160918724# disk.dm-0.out‚12119.96142578125 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0* proc.memory.percent‚16.709327697753906 network.recv‚ 1.010074e+06$ memory_percent‚96.97198867797852 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 262.50390625 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚588.9651336669922 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 6.684055e+06 disk.dm-0.in‚ 5889.37890625 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚93.19775390625& disk.nvme0n1.out‚12119.96142578125 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2737.65625) disk./.usagePercent‚39.52594429505438 disk.dm-1.out‚ 262.50390625& proc.memory.availableMB‚ 496.109375\dùÔ“Ó‚(:ˆ ‹âìÒ”º˜º gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÖΦóÔ‚(:è ‹âìÒñÝм disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125( disk.nvme0c0n1.out‚12125.32080078125# memory_percent‚96.9705581665039 network.sent‚ 6.686914e+06 disk.sda2.in‚0 disk.drbd2165.in‚0# disk.dm-0.out‚12125.32080078125# disk.nvme0c0n1.in‚ 5889.37890625! disk.nvme0n1.in‚ 5889.37890625 cpu‚26.176155628371454 proc.cpu.threads‚17 network.recv‚ 1.012738e+06& disk.nvme0n1.out‚12125.32080078125 disk.sda.out‚ 262.8828125 disk.sda1.out‚0$ disk./.usageGB‚588.9650764465332 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 proc.memory.rssMB‚ 2737.65625* proc.memory.percent‚16.709327697753906) disk./.usagePercent‚39.52594045494136 disk.sda.in‚ 0.00390625 disk.drbd2165.out‚0 disk.rbd1.out‚93.27587890625 disk.rbd0.in‚0 disk.dm-0.in‚ 5889.37890625 disk.dm-1.out‚ 262.8828125% proc.memory.availableMB‚ 496.34375 disk.sda3.out‚ 262.8828125 yØ2Õ- _wandbruntime‚3726  _runtime‚3726 êMõÉÖà ' lr-AdamW/pg1‚0.00039286484444113105 ' lr-AdamW/pg2‚0.00039286484444113105 trainer/global_step‚4349 % _timestamp‚1.7843612339638772e+09 _runtime‚3726.039145819 _step‚354ù»ûžÎ×È trainer/global_step‚4349 % _timestamp‚1.7843612339638772e+09 _runtime‚3726.039145819 _step‚354 ' lr-AdamW/pg1‚0.00039286484444113105 ' lr-AdamW/pg2‚0.00039286484444113105â‹]¦2Ø- _wandbruntime‚3726  _runtime‚3726´€ÿá'ÙÛO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚659829969346560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656443338600448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4581422253918 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4581762803808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150337109947392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16952078 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664418759436288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4638929087488 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34409720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664418759436288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401057792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237369 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4588773050368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661024743815006 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74657068 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7621831 % _timestamp‚1.7843612339646585e+09 _runtime‚3726.040397863 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656443321561088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588431711232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150337231816704 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401057792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34409720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40247346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401057792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34409720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40247346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4581422253918 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4581762803808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150337109947392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16952078 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664418759436288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4638929087488 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34409720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664418759436288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401057792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237369 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4588773050368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661024743815006 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74657068 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7621831 % _timestamp‚1.7843612339646585e+09 _runtime‚3726.040397863 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656443321561088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588431711232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150337231816704 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663773 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4589444338176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40256412 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 3726.65323778 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚659977227415552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚659977210376192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4582433307640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16955599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚4349 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7623573 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129962 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663773 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664567013100032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150371674807808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34417354 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4589444338176 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7623576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2"Æ}²î&ÝèM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663773 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673766 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4589444338176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40256412 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 3726.65323778 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚659977227415552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚659977210376192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4582433307640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16955599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚4349 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7623573 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129962 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663773 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664567013100032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150371674807808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34417354 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4589444338176 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7623576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34417352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4640046577152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269äcco32Þ- _wandbruntime‚3726  _runtime‚3726$ì/ ߃ 4loss/FBt1/assignment_loss‚0.00016225435587950834 /loss/FRt1/detection_loss‚0.5402942279285337 .loss/FRt2/detection_loss‚1.502950339436772 epoch‚10 1loss/SRqqt2/detection_loss‚0.6836802508122948 /loss/FBt1/detection_loss‚0.5311151419976546 /loss/FBt2/detection_loss‚0.5325709037944734 $loss/total_loss‚8.46622820042547 0loss/FRt1/assignment_loss‚1.7797216489487673 2loss/SRqqt2/assignment_loss‚0.5204293973538555 1loss/FBt2/assignment_loss‚0.06311264338909146 1loss/SRqqt1/detection_loss‚1.0689804602347845 trainer/global_step‚4349 % _timestamp‚1.7843612345797844e+09 _runtime‚3726.654847919 _step‚357 0loss/FRt2/assignment_loss‚0.7038718051281735 2loss/SRqqt1/assignment_loss‚0.5393391270451892ÆÜßàˆ 2loss/SRqqt2/assignment_loss‚0.5204293973538555 1loss/FBt2/assignment_loss‚0.06311264338909146 1loss/SRqqt1/detection_loss‚1.0689804602347845 trainer/global_step‚4349 % _timestamp‚1.7843612345797844e+09 _runtime‚3726.654847919 _step‚357 0loss/FRt2/assignment_loss‚0.7038718051281735 2loss/SRqqt1/assignment_loss‚0.5393391270451892 4loss/FBt1/assignment_loss‚0.00016225435587950834 /loss/FRt1/detection_loss‚0.5402942279285337 .loss/FRt2/detection_loss‚1.502950339436772 epoch‚10 1loss/SRqqt2/detection_loss‚0.6836802508122948 /loss/FBt1/detection_loss‚0.5311151419976546 /loss/FBt2/detection_loss‚0.5325709037944734 $loss/total_loss‚8.46622820042547 0loss/FRt1/assignment_loss‚1.7797216489487673åÒ×]C“á‚(:ˆ šâìÒ¦­³½ gpu.0.gpu‚46.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>§%îâ‚(:ã šâìÒØÁò¾ disk.sda3.out‚ 263.33203125 disk.dm-0.in‚ 5889.37890625 disk.dm-0.out‚12128.88671875 cpu‚26.175619405898743 disk.rbd1.in‚ 0.01953125# disk.nvme0c0n1.in‚ 5889.37890625 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.019256e+06% disk.nvme0c0n1.out‚12128.88671875 proc.memory.rssMB‚ 2737.65625! disk.nvme0n1.in‚ 5889.37890625' proc.memory.availableMB‚ 496.4140625# disk./.usageGB‚588.965648651123 disk.sda.out‚ 263.33203125 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.00390625 proc.cpu.threads‚17 network.sent‚ 6.816849e+06# disk.nvme0n1.out‚12128.88671875$ memory_percent‚96.97012901306152* proc.memory.percent‚16.709327697753906( disk./.usagePercent‚39.5259788560715 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 263.33203125 disk.rbd1.out‚93.27587890625 disk.rbd0.out‚0xŽìlžã‚(:“ ©âìÒ ”º gpu.0.gpu‚54.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚211.465' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.293000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t—ñïKä‚(:ë ©âìÒ†Š¼ disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0" disk.dm-0.out‚12136.0419921875 proc.cpu.threads‚17 network.sent‚ 6.820608e+06$ disk./.usageGB‚588.9655380249023' proc.memory.availableMB‚ 496.2734375 disk.rbd0.out‚0! disk.nvme0n1.in‚ 5934.92578125 disk.sda1.YM"‘«in‚0 network.recv‚ 1.022761e+06 disk.dm-0.in‚ 5934.92578125 proc.memory.rssMB‚ 2737.65625 disk.rbd1.out‚93.33837890625% disk.nvme0n1.out‚12136.0419921875 disk.sda.in‚ 0.00390625* proc.memory.percent‚16.709327697753906 disk.rbd1.in‚ 0.01953125 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 263.49609375 disk.sda1.out‚0$ memory_percent‚96.97098731994629 cpu‚26.175095540244833) disk./.usagePercent‚39.52597143185301# disk.nvme0c0n1.in‚ 5934.92578125 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625' disk.nvme0c0n1.out‚12136.0419921875 disk.sda.out‚ 263.49609375 disk.drbd2165.out‚0 disk.dm-1.out‚ 263.49609375Ü´›þå‚(:’ ¸âìÒꊜº gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~-è6õæ‚(:ê ¸âìÒ¦‚Õ¼ disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625* proc.memory.percent‚16.709327697753906 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0 disk.sda.in‚ 0.00390625 disk.sda.out‚ 263.83203125 disk.sda2.in‚0 disk.dm-1.out‚ 263.83203125 disk.drbd2165.out‚0 disk.dm-0.in‚ 5934.92578125" disk.dm-0.out‚12140.9111328125% disk.nvme0n1.out‚12140.9111328125 disk.sda1.in‚0! disk.nvme0n1.in‚ 5934.92578125( proc.memory.availableMB‚ 496.47265625 proc.cpu.threads‚17 network.recv‚ 1.025983e+06 disk.sda2.out‚0' disk.nvme0c0n1.out‚12140.9111328125 disk.sda3.out‚ 263.83203125 cpu‚26.17452487285389 network.sent‚ 6.82686e+06$ disk./.usageGB‚588.9654655456543 disk.rbd1.out‚93.33837890625# disk.nvme0c0n1.in‚ 5934.92578125 disk.sda1.out‚0$ memory_percent‚96.96977138519287 proc.memory.rssMB‚ 2737.65625 disk.rbd1.in‚ 0.01953125) disk./.usagePercent‚39.52596656770986 disk.drbd2165.in‚0‚ô2ç- _wandbruntime‚3766  _runtime‚3766‚†l£ÇèÁ % _timestamp‚1.7843612747417812e+09 _runtime‚3766.820418074 _step‚358 & lr-AdamW/pg1‚0.0003918281101472575 & lr-AdamW/pg2‚0.0003918281101472575 trainer/global_step‚4399©gCÌéÆ % _timestamp‚1.7843612747417812e+09 _runtime‚3766.820418074 _step‚358 & lr-AdamW/pg1‚0.0003918281101472575 & lr-AdamW/pg2‚0.0003918281101472575 trainer/global_step‚4399æ•v¸g2ê- _wandbruntime‚3766  _runtime‚3766^8rà'ëÚO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4690026978816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148041216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7709078 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7709081 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40701798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147369637398528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚667192037867520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚671831506419712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4639468552192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152059645765120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚4399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚3766.824019603 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4690111361536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663768872194048Š”å'ìßO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4639468552192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493120 % _timestamp‚1.7843612747485232e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚667192020828160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34791320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9432987 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4639127213056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚671831148041216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34791322 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚359 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17142065 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17142331 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668400893895514 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632379290716 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639127213056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663768855154688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚667192020828160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148041216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7709078 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7709081 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40701798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147369637398528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚667192037867520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚671831506419712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4639468552192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152059645765120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚4399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚3766.824019603 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4690111361536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663768872194048 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4690026978816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4691133728768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210470 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499825 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139840000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668548415621108 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671979760083456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667339278897152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚671979760083456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9435288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚4399 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40710864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509818 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40700873 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17145850 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667339261857792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139840000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7710826 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169÷ì…¥ï&ïéM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40710864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509818 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40700873 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17145850 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667339261857792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139840000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7710826 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4691133728768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210470 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499825 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139840000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668548415621108 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671979760083456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667339278897152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚671979760083456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9435288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚4399èA”A‰2ð- _wandbruntime‚3767  _runtime‚3767ÓU†Öñþ 0loss/FRt1/assignment_loss‚1.7440312695116855 2loss/SRqqt2/assignment_loss‚0.5192025934109648 /loss/FRt2/detection_loss‚1.5381162174806395 epoch‚10 2loss/FBt1/assignment_loss‚0.029361371683118403 .loss/FRt1/detection_loss‚0.568540556250532 1loss/SRqqt2/detection_loss‚0.6502236619541476 trainer/global_step‚4399 2loss/SRqqt1/assignment_loss‚0.6563146205449142 1loss/SRqqt1/detection_loss‚0.9900390286439735 /loss/FBt1/detection_loss‚0.6819376986473799 /loss/FBt2/detection_loss‚0.5877023159914981 0loss/FRt2/assignment_loss‚0.7212398682815441 1loss/FBt2/assignment_loss‚0.03151435330291108 $loss/total_loss‚8.71822355570331 $ _timestamp‚1.784361275362473e+09 _runtime‚ 3767.4375465 _step‚3614æ òƒ 0loss/FRt2/assignment_loss‚0.7212398682815441 1loss/FBt2/assignment_loss‚0.03151435330291108 $loss/total_loss‚8.71822355570331 $ _timestamp‚1.784361275362473e+09 _runtime‚ 3767.4375465 _step‚361 0loss/FRt1/assignment_loss‚1.7440312695116855 2loss/SRqqt2/assignment_loss‚0.5192025934109648 /loss/FRt2/detection_loss‚1.5381162174806395 epoch‚10 2loss/FBt1/assignment_loss‚0.029361371683118403 .loss/FRt1/detection_loss‚0.568540556250532 1loss/SRqqt2/detection_loss‚0.6502236619541476 trainer/global_step‚4399 2loss/SRqqt1/assignment_loss‚0.6563146205449142 1loss/SRqqt1/detection_loss‚0.9900390286439735 /loss/FBt1/detection_loss‚0.6819376986473799 /loss/FBt2/detection_loss‚0.5877023159914981é†é@‘ó‚(:† ÇâìÒ—ÜÛ» gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í`ðô‚(:å ÇâìÒÑ´½$ disk./.usageGB‚588.9655494689941 disk.rbd1.out‚93.41650390625 disk.sda.in‚ 0.00390625( proc.memory.availableMB‚ 495.21484375 proc.cpu.threads‚17 network.recv‚ 1.029615e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.71736240386963) disk./.usagePercent‚39.52597219987561 disk.sda1.in‚0 disk.rbd0.out‚0$ memory_percent‚96.97744846343994 disk.dm-1.in‚ 0.00390625% disk.nvme0c0n1.out‚12146.53515625 disk.sda.out‚ 264.24609375 disk.drbd2165.out‚0 disk.dm-0.in‚ 5934.92578125 disk.rbd1.in‚ 0.01953125# disk.nvme0c0n1.in‚ 5934.92578125# proc.memory.rssMB‚ 2738.97265625 disk.sda1.out‚0# disk.nvme0n1.out‚12146.53515625 disk.sda3.in‚ 0.00390625 network.sent‚ 6.831997e+06 disk.dm-0.out‚12146.53515625! disk.nvme0n1.in‚ 5934.92578125 cpu‚26.182566050362 disk.dm-1.out‚ 264.24609375 disk.rbd0.in‚0 disk.sda3.out‚ 264.246093753'Æ 2õ- _wandbruntime‚3786  _runtime‚3786¦­=Í­=ö§{ :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20281226933002472 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984380602836609 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8393046259880066 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21276895701885223 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6573731303215027 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.749640941619873 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.359375 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18411053717136383 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959702491760254 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 'particle/f_score‚0.6406281590461731 epoch‚10 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6512831449508667 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032083503901958466 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _runtime‚3786.173421561 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4655686020851135 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17808738350868225 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6271027326583862 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9890236854553223 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23273181915283203 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 % _timestamp‚1.7843612940974941e+09 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.637614369392395 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6366801261901855 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5745698809623718 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.751302182674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6283671259880066 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027754046022892 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975327849388123 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025474190711975098 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8251259922981262 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4987974166870117 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2045810967683792 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 _step‚362 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2334856241941452 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1843705028295517 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6545432806015015 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950778484344482 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6567795276641846 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6434842944145203 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961156249046326 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1840527355670929 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8224301338195801 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7671142816543579 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029356280341744423 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7328484654426575 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20661653578281403 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027309536933898926 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026201900094747543 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3892938494682312 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8694630861282349 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6716184616088867 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7052015662193298 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5654491186141968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6381706595420837 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16698595881462097 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7525990009307861 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22179542481899261 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6816026568412781 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_3‚0.9280992150306702 *jet/accuracy_1_of_4‚0.9817814826965332 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6130393743515015 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23216690123081207 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2203969955444336 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20134301483631134 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.466803640127182 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890558123588562 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_4‚0.8347601294517517 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5839284062385559 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1970001608133316 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7782639265060425 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029356280341744423 particle/accuracy_1_of_2‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7333527207374573 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.061386626213788986 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_2‚0.7911761999130249 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19751545786857605 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6992307305335999 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7535815238952637 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6808972954750061 +particle/specificity‚0.9095831513404846 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965940117835999 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2845022678375244 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5730521082878113 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8721848726272583 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953246116638184 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5400000214576721 particle/accuracy_1_of_3‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2202485352754593 +jet/accuracy_3_of_3‚0.07372625917196274 (particle/accuracy‚0.8489196300506592 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8809758424758911 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7548105120658875 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.869255542755127 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11699272692203522 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1341666728258133 jet/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7081871628761292 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6484375 jet/accuracy_3_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6544588208198547 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8230372667312622 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3860423266887665 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03287501260638237 *jet/accuracy_1_of_1‚0.3316929340362549 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942343831062317 .particle/accuracy_3_of_4‚0.997619092464447 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23959007859230042 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11456790566444397 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3248765468597412 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_3_of_3‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2202485352754593 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4808747470378876 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9433409571647644 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8704837560653687 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8721848726272583 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7599501013755798 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7567905783653259 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.047294773161411285 particle/accuracy_5_of_5‚0 trainer/global_step‚4399 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977334141731262 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 )loss/val_total_loss‚8.544140435757468 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7393327355384827 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13218005001544952 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975327849388123 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8461261987686157 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28317785263061523 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22049109637737274 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.779434084892273 +jet/accuracy_3_of_4‚0.08423537015914917 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6567795276641846 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6973555684089661 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4523448348045349 6validation_average_jet_accuracy‚0.3743986487388611 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8984297513961792 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3215949535369873 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04521489888429642 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024638459086418152 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6605957746505737 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09968820214271545 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7618591785430908 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.15296313166618347 /particle/accuracy_4_of_4‚0.9362794756889343 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48362940549850464 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6785290837287903 ("Purity/2FRt1SRqqt1FBt/event_purity‚0â9†Ê ÷¬{ 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18411053717136383 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959702491760254 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 'particle/f_score‚0.6406281590461731 epoch‚10 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6512831449508667 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032083503901958466 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _runtime‚3786.173421561 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4655686020851135 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17808738350868225 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6271027326583862 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9890236854553223 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23273181915283203 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 % _timestamp‚1.7843612940974941e+09 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.637614369392395 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6366801261901855 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5745698809623718 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.751302182674408 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6283671259880066 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.027754046022892 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975327849388123 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025474190711975098 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8251259922981262 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4987974166870117 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2045810967683792 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 _step‚362 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2334856241941452 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1843705028295517 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6545432806015015 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950778484344482 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6567795276641846 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqtjÜ Ýè/1FBt/FRt_purity‚0.6434842944145203 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961156249046326 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1840527355670929 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8224301338195801 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7671142816543579 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029356280341744423 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7328484654426575 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20661653578281403 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027309536933898926 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026201900094747543 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3892938494682312 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8694630861282349 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6716184616088867 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7052015662193298 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5654491186141968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6381706595420837 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16698595881462097 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7525990009307861 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22179542481899261 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6816026568412781 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_3‚0.9280992150306702 *jet/accuracy_1_of_4‚0.9817814826965332 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6130393743515015 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23216690123081207 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2203969955444336 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20134301483631134 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.466803640127182 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9890558123588562 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_4‚0.8347601294517517 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5839284062385559 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1970001608133316 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7782639265060425 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029356280341744423 particle/accuracy_1_of_2‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7333527207374573 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.061386626213788986 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_2‚0.7911761999130249 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19751545786857605 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6992307305335999 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7535815238952637 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6808972954750061 +particle/specificity‚0.9095831513404846 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965940117835999 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2845022678375244 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5730521082878113 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8721848726272583 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953246116638184 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5400000214576721 particle/accuracy_1_of_3‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2202485352754593 +jet/accuracy_3_of_3‚0.07372625917196274 (particle/accuracy‚0.8489196300506592 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8809758424758911 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7548105120658875 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.869255542755127 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11699272692203522 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1341666728258133 jet/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7081871628761292 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6484375 jet/accuracy_3_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6544588208198547 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8230372667312622 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3860423266887665 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03287501260638237 *jet/accuracy_1_of_1‚0.3316929340362549 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942343831062317 .particle/accuracy_3_of_4‚0.997619092464447 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23959007859230042 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11456790566444397 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3248765468597412 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_3_of_3‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2202485352754593 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4808747470378876 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9433409571647644 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8704837560653687 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8721848726272583 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7599501013755798 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7567905783653259 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.047294773161411285 particle/accuracy_5_of_5‚0 trainer/global_step‚4399 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977334141731262 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 )loss/val_total_loss‚8.544140435757468 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7393327355384827 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13218005001544952 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975327849388123 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8461261987686157 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28317785263061523 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22049109637737274 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.779434084892273 +jet/accuracy_3_of_4‚0.08423537015914917 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6567795276641846 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6973555684089661 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4523448348045349 6validation_average_jet_accuracy‚0.3743986487388611 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8984297513961792 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3215949535369873 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04521489888429642 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.024638459086418152 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6605957746505737 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09968820214271545 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7618591785430908 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.15296313166618347 /particle/accuracy_4_of_4‚0.9362794756889343 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48362940549850464 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6785290837287903 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20281226933002472 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984380602836609 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8393046259880066 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21276895701885223 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6573731303215027 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.749640941619873 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.359375ê›Ý݈‘ø‚(:† ÖâìÒûØ—» gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚261.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–þ¶èüù‚(:ñ ÖâìÒÀ—Œ½ network.sent‚ 6.894885e+06 network.recv‚ 1.033817e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 5934.92578125$ memory_percent‚97.00486660003662* proc.memory.percent‚16.741514205932617 disk.rbd1.in‚ 0.01953125# disk.nvme0c0n1.in‚ 5934.92578125 disk.sda.in‚ 0.00390625 proc.cpu.threads‚17$ disk./.usageGB‚588.9655380249023 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚12150.59326171875 disk.sda.out‚ 264.68359375& disk.nvme0n1.out‚12150.59326171875" proc.memory.rssMB‚ 2742.9296875 cpu‚26.186369954084743) disk./.usagePercent‚39.52597143185301 disk.rbd0.in‚0( proc.memory.availableMB‚ 490.72265625 disk.rbd1.out‚93.41650390625! disk.nvme0n1.in‚ 5934.92578125 disk.sda3.out‚ 264.68359375 disk.dm-1.in‚ 0.00390625( disk.nvme0c0n1.out‚12150.59326171875 disk.sda1.out‚0 disk.dm-1.out‚ 264.68359375ÿ^£È“ú‚(:ˆ åâìÒÜ»‘º gpu.0.gpu‚54.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚120.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S _–õû‚(:ê åâìÒÙÜ ½ disk.rbd1.out‚93.47900390625 cpu‚26.185573559875483) disk./.usagePercent‚39.52598013610917$ memory_percent‚97.00336456298828" proc.memory.rssMB‚ 2742.9296875 disk.rbd0.in‚0# disk.dm-0.out‚12157.25439453125 disk.drbd2165.in‚0% proc.memory.availableMB‚ 490.96875 network.sent‚ 6.95211e+06 disk.sda3.out‚ 265.1796875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.038154e+06# disk.nvme0c0n1.in‚ 5982.42578125! disk.nvme0n1.in‚ 5982.42578125& disk.nvme0n1.out‚12157.25439453125 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 5982.42578125 disk.dm-1.out‚ 265.1796875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12157.25439453125 disk.sda.out‚ 265.1796875 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625* proc.memory.percent‚16.741514205932617 disk.sda2.in‚0$ disk./.usageGB‚588.9656677246094 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125๼7œü‚(:‘ ôâìÒÈž» gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0  ;»üý‚(:ñ ôâìÒº‚½$ disk./.usageGB‚588.9656791687012 disk.sda.out‚ 265.48046875 disk.sda2.in‚0 disk.rbd1.in‚ 0.01953125! disk.nvme0n1.in‚ 5982.42578125 proc.cpu.threads‚17 disk.sda1.out‚0# disk.dm-0.out‚12161.24560546875 disk.dm-1.out‚ 265.48046875$ memory_percent‚97.00520038604736 disk.sda2.out‚0 disk.drbd2165.out‚0 network.sent‚ 6.957094e+06 network.recv‚ 1.041728e+06 disk.sda.in‚ 0.00390625 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 265.48046875" proc.memory.rssMB‚ 2742.9296875# disk.nvme0c0n1.in‚ 5982.42578125( proc.memory.availableMB‚ 490.66796875 disk.drbd2165.in‚0 disk.dm-0.in‚ 5982.42578125( disk.nvme0c0n1.out‚12161.24560546875& disk.nvme0n1.out‚12161.24560546875 disk.rbd1.out‚93.47900390625 disk.rbd0.in‚0) disk./.usagePercent‚39.52598090413177* proc.memory.percent‚16.741514205932617 cpu‚26.185113543675453 disk.rbd0.out‚0¥‚§2þ- _wandbruntime‚3826  _runtime‚3826©X¦ÇÿÁ _step‚363 & lr-AdamW/pg1‚0.0003907801790774355 & lr-AdamW/pg2‚0.0003907801790774355 trainer/global_step‚4449 % _timestamp‚1.7843613344188054e+09 _runtime‚3826.494071665ÜÁ]mÌ€Æ & lr-AdamW/pg1‚0.0003907801790774355 & lr-AdamW/pg2‚0.0003907801790774355 trainer/global_step‚4449 % _timestamp‚1.7843613344188054e+09 _runtime‚3826.494071665 _step‚363ëmÛFõ2- _wandbruntime‚3826  _runtime‚3826!Ûà'‚ÚO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681191553259008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676483638960128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9567505 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚3826.495074238 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41216595 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149056968812544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4759316823040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153816285635584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677703784504581 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673003069001728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502853 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35293586 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35293588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7798848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681191194880512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7798851 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41226572 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚676483655999488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4707897259520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35293586 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4759401205760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚676483638960128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194880512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673003051962368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4707897259520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976±n]9ƒßO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚3826.495074238 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41216595 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149056968812544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4759316823040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153816285635584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677703784504581 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673003069001728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502853 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35293586 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35293588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7798848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681191194880512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7798851 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41226572 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚676483655999488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4707897259520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35293586 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4759401205760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚676483638960128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194880512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673003051962368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4707897259520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681191553259008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676483638960128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9567505ì!I82„- _wandbruntime‚3827  _runtime‚3827†MÇê&…äM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4760427111936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4708568547328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149090416183296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4701386006685 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41235638 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚677850948640925 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚365 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35301220 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676630879989760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9569295 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17369888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843613350341294e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673149579673600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4701726556687 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚676630897029120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9569550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4760342736384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681339806922752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4708568547328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681339448537088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4708909893632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚3827.109773451 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41225647 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41235638 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4708568547328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149090416183296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4701386006685 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41235638 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚677850948640925 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚365 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35301220 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676630879989760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9569295 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17369888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843613350341294e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673149579673600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4701726556687 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚676630897029120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9569550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4760342736384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681339806922752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4708568547328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681339448537088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4708909893632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚3827.109773451íU>r2‡- _wandbruntime‚3827  _runtime‚3827<²å6 ˆƒ _runtime‚3827.111320601 /loss/FRt1/detection_loss‚0.5519188725482588 _step‚366 0loss/FRt2/assignment_loss‚0.7392934018507936 2loss/SRqqt1/assignment_loss‚0.5768258457372238 2loss/SRqqt2/assignment_loss‚0.5331107495931882 2loss/FBt2/assignment_loss‚0.014685195944902018 1loss/SRqqt2/detection_loss‚0.6537463626898877 0loss/FRt1/assignment_loss‚1.8537087982147231 1loss/SRqqt1/detection_loss‚1.0079485870927283 /loss/FBt1/detection_loss‚0.8141398555325234 /loss/FBt2/detection_loss‚0.5065424980690879 %loss/total_loss‚8.845484598384157 epoch‚11 2loss/FBt1/assignment_loss‚0.012959944852379834 /loss/FRt2/detection_loss‚1.5806044862584625 trainer/global_step‚4449 $ _timestamp‚1.784361335036268e+09Ô‰ˆ /loss/FRt2/detection_loss‚1.5806044862584625 trainer/global_step‚4449 $ _timestamp‚1.784361335036268e+09 _runtime‚3827.111320601 /loss/FRt1/detection_loss‚0.5519188725482588 _step‚366 0loss/FRt2/assignment_loss‚0.7392934018507936 2loss/SRqqt1/assignment_loss‚0.5768258457372238 2loss/SRqqt2/assignment_loss‚0.5331107495931882 2loss/FBt2/assignment_loss‚0.014685195944902018 1loss/SRqqt2/detection_loss‚0.6537463626898877 0loss/FRt1/assignment_loss‚1.8537087982147231 1loss/SRqqt1/detection_loss‚1.0079485870927283 /loss/FBt1/detection_loss‚0.8141398555325234 /loss/FBt2/detection_loss‚0.5065424980690879 %loss/total_loss‚8.845484598384157 epoch‚11 2loss/FBt1/assignment_loss‚0.012959944852379834îyߟ7“Š‚(:ˆ ƒãìÒÜí¶º gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{pÀ/ô‹‚(:é ƒãìÒÝÕÚ¼! disk.nvme0n1.in‚ 5982.42578125 disk.sda.out‚ 265.62109375 disk.sda3.out‚ 265.62109375) disk./.usagePercent‚39.52597936808657 disk.rbd1.out‚93.55712890625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 490.6484375 cpu‚26.18420530232389 disk.dm-1.in‚ 0.00390625$ disk.nvme0n1.out‚12168.138671875" proc.memory.rssMB‚ 2742.9296875* proc.memory.percent‚16.741514205932617 network.sent‚ 6.962235e+06 disk.sda2.out‚0! disk.dm-0.out‚12168.138671875 disk.dm-1.out‚ 265.62109375# disk.nvme0c0n1.in‚ 5982.42578125& disk.nvme0c0n1.out‚12168.138671875 disk.sda3.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 1.045305e+06 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚97.00531959533691$ disk./.usageGB‚588.9656562805176 disk.dm-0.in‚ 5982.42578125 disk.rbd1.in‚ 0.01953125³ ¬’Œ‚(:‡ ’ãìÒ¡Éʺ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í^åVö‚(:ë ’ãìÒ¦ªç¼' proc.memory.availableMB‚ 490.6328125 network.sent‚ 7.025305e+06 network.recv‚ 1.049725e+06 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 5982.42578125 disk.rbd0.out‚0! disk.nvme0n1.in‚ 5982.42578125 disk.sda.out‚ 266.140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚93.55712890625 disk.sda3.out‚ 266.140625$ memory_percent‚97.00541496276855* disk./.usagePercent‚39.525928166579725 disk.sda.in‚ 0.00390625" proc.memory.rssMB‚ 2742.9296875 disk.sda3.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625 disk.drbd2165.out‚0# disk.dm-0.out‚12173.71923828125( disk.nvme0c0n1.out‚12173.71923828125 proc.cpu.threads‚17$ disk./.usageGB‚588.9648933410645 disk.dm-1.out‚ 266.140625 disk.rbd1.in‚ 0.01953125# disk.nvme0c0n1.in‚ 5982.42578125 cpu‚26.183942601165388 disk.sda2.in‚0* proc.memory.percent‚16.741514205932617& disk.nvme0n1.out‚12173.71923828125ç½>*2Ž- _wandbruntime‚3867  _runtime‚3867~g}ÇÁ & lr-AdamW/pg1‚0.0003897211185383195 & lr-AdamW/pg2‚0.0003897211185383195 trainer/global_step‚4499 % _timestamp‚1.7843613751628122e+09 _runtime‚3867.241402034 _step‚367yì}ÌÆ & lr-AdamW/pg1‚0.0003897211185383195 & lr-AdamW/pg2‚0.0003897211185383195 trainer/global_step‚4499 % _timestamp‚1.7843613751628122e+09 _runtime‚3867.241402034 _step‚367ïÍ 2‘- _wandbruntime‚3867  _runtime‚3867óyþŠ ’ØO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941863936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.³BÎbTall.freed‚17556968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚680328585555968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683845690441728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35675188 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683845690441728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚688604300242432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚680328602595328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4758251422208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17557238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685079934585089 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4750991439871 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3867.242888949 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4810690324992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155539115676160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683845707481088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9670870 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683845707481088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4758592761344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35675186 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4751331989761 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7886101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4810603845120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚685079576995839 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7886098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150728425351168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77356212 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9671137 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888˜“­Mã'“ÝO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41681024 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688604300242432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4758251422208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41671047 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941863936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17556968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚680328585555968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683845690441728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35675188 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683845690441728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚688604300242432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚680328602595328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4758251422208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17557238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685079934585089 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4750991439871 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3867.242888949 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4810690324992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155539115676160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683845707481088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9670870 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683845707481088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4758592761344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35675186 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4751331989761 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7886101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4810603845120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚685079576995839 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7886098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150728425351168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77356212 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9671137 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35682820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4811618871296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4752343043595 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688752553906176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150761835497472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4811709538304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683992931471360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4759605395456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683992931471360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688752195520512 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7887846 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚4499 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4752002493593 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41690090 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚680475113267200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17560799 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685227098721433 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚685227456310795 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35682820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155573563648000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35682818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41690090 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41680099 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683992948510720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9673213 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9672956 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚369 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4759264049152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚3867.854341782 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150761854109696éWO+Š–éM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35682820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4811618871296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4752343043595 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688752553906176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150761835497472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4811709538304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683992931471360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4759605395456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683992931471360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688752195520512 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7887846 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚4499 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4752002493593 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41690090 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚680475113267200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17560799 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685227098721433 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚685227456310795 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35682820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155573563648000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35682818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41690090 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41680099 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683992948510720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9673213 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9672956 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚369 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4759264049152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚3867.854341782 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150761854109696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 Eœ2—- _wandbruntime‚3867  _runtime‚3867E{˜Ð˜‚ 0loss/FRt1/assignment_loss‚1.8156507521981502 0loss/FBt2/assignment_loss‚0.0099987219254403 /loss/FBt1/detection_loss‚0.8236390455595909 epoch‚11 _runtime‚3867.856340025 0loss/FRt2/assignment_loss‚0.7954418282463865 3loss/FBt1/assignment_loss‚0.0004574455133479205 .loss/FRt1/detection_loss‚0.519016324077895 1loss/SRqqt2/detection_loss‚0.6058330860747854 _step‚370 2loss/SRqqt2/assignment_loss‚0.5103136667542164 /loss/FBt2/detection_loss‚0.4509762608910953 % _timestamp‚1.7843613757812953e+09 2loss/SRqqt1/assignment_loss‚0.5495873892298181 /loss/FRt2/detection_loss‚1.4943100122827158 1loss/SRqqt1/detection_loss‚0.8951097927056253 %loss/total_loss‚8.470334325459067 trainer/global_step‚4499Ýûü ™‡ 1loss/SRqqt1/detection_loss‚0.8951097927056253 %loss/total_loss‚8.470334325459067 trainer/global_step‚4499 0loss/FRt1/assignment_loss‚1.8156507521981502 0loss/FBt2/assignment_loss‚0.0099987219254403 /loss/FBt1/detection_loss‚0.8236390455595909 epoch‚11 _runtime‚3867.856340025 0loss/FRt2/assignment_loss‚0.7954418282463865 3loss/FBt1/assignment_loss‚0.0004574455133479205 .loss/FRt1/detection_loss‚0.519016324077895 1loss/SRqqt2/detection_loss‚0.6058330860747854 _step‚370 2loss/SRqqt2/assignment_loss‚0.5103136667542164 /loss/FBt2/detection_loss‚0.4509762608910953 % _timestamp‚1.7843613757812953e+09 2loss/SRqqt1/assignment_loss‚0.5495873892298181 /loss/FRt2/detection_loss‚1.4943100122827158òóØ©áš‚(:’ ¡ãìÒÝ©Ä» gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚245.093' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚49.01859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06(:ä ¡ãìÒʼnç¼ disk.sda3.out‚266.6875$ memory_percent‚97.00379371643066* disk./.usagePercent‚39.525932774715336$ disk./.usageGB‚588.9649620056152 disk.sda1.out‚0# disk.dm-0.out‚12180.38330078125 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚93.68212890625 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 6039.08203125 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 6039.08203125* proc.memory.percent‚16.741514205932617 proc.cpu.threads‚17 network.sent‚ 7.030431e+06 disk.drbd2165.out‚0 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01953125& disk.nvme0n1.out‚12180.38330078125 cpu‚26.18368363849441 disk.sda1.in‚0 disk.dm-0.in‚ 6039.08203125 disk.sda.in‚ 0.00390625' proc.memory.availableMB‚ 490.8984375 disk.dm-1.out‚266.6875( disk.nvme0c0n1.out‚12180.38330078125" proc.memory.rssMB‚ 2742.9296875 network.recv‚ 1.053327e+06 disk.sda.out‚266.6875 Lìãžœ‚(:“ °ãìÒ䯎» gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚252.924' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.584799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=¦ŠÝñ‚(:æ °ãìÒÞο½ disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.drbd2165.out‚0 network.recv‚ 1.057608e+06 disk.sda3.out‚267.0625# disk.dm-0.out‚12184.38330078125 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚93.68603515625 disk.sda.out‚267.0625 disk.rbd0.in‚0 cpu‚26.182713654091202 disk.drbd2165.in‚0 disk.dm-1.out‚267.0625 disk.rbd1.in‚ 0.01953125" proc.memory.rssMB‚ 2742.9296875$ disk./.usageGB‚588.9649353027344 disk.dm-0.in‚ 6039.08203125 disk.rbd0.out‚0! disk.nvme0n1.in‚ 6039.08203125( proc.memory.availableMB‚ 490.83203125 network.sent‚ 7.093306e+06 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6039.08203125( disk.nvme0c0n1.out‚12184.38330078125& disk.nvme0n1.out‚12184.38330078125$ memory_percent‚97.00419902801514 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625* disk./.usagePercent‚39.525930982662594 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.741514205932617õ…ZK“ž‚(:ˆ ¿ãìÒ–êöº gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Btq럂(:à ¿ãìÒÙˆí¼$ memory_percent‚97.00450897216797* disk./.usagePercent‚39.525916646240674 disk.rbd1.in‚ 0.01953125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 6039.08203125 network.sent‚ 7.098166e+06! disk.nvme0n1.in‚ 6039.08203125 cpu‚26.182406138238623& disk.nvme0n1.out‚12190.24755859375 disk.sda.in‚ 0.00390625 disk.sda.out‚267.375 disk.sda2.in‚0$ disk./.usageGB‚588.9647216796875 disk.sda3.out‚267.375* proc.memory.percent‚16.741514205932617 proc.cpu.threads‚17 disk.rbd1.out‚93.76416015625# disk.nvme0c0n1.in‚ 6039.08203125 disk.dm-1.in‚ 0.00390625 network.recv‚ 1.060578e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12190.24755859375 disk.sda2.out‚0% proc.memory.availableMB‚ 490.78125" proc.memory.rssMB‚ 2742.9296875 disk.drbd2165.out‚0# disk.dm-0.out‚12190.24755859375 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.dm-1.out‚267.375­AH®2 - _wandbruntime‚3908  _runtime‚3908Ã"Õ}É¡Ã % _timestamp‚1.7843614161438162e+09 _runtime‚3908.219183853 _step‚371 ' lr-AdamW/pg1‚0.00038865099655138876 ' lr-AdamW/pg2‚0.00038865099655138876 trainer/global_step‚4549ü«ñ¥Î¢È ' lr-AdamW/pg2‚0.00038865099655138876 trainer/global_step‚4549 % _timestamp‚1.7843614161438162e+09 _runtime‚3908.219183853 _step‚371 ' lr-AdamW/pg1‚0.00038865099655138876óÙ6Ñä2£- _wandbruntime‚3908  _runtime‚3908Ìù€Æà'¤ÚO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36056788 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692455727076347 trainer/global_step‚4549 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9774767 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696016688847360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36056788 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17747861 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4861696572416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691207741923328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36056786 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7973348 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157261650524160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4808946924032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚691207741923328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚3908.220272588 _step‚372 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691207758962688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚687654136188928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696017047225856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4809288263168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36056786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4861787246592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688847360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4808946924032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152399844665344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚687654119149568 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42125499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692456084665597 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4801607926779 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚696017047225856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152399863277568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42125499 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42135476 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17748118 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9774513 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4801948476669 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 % _timestamp‚1.7843614161447575e+09­9%jå'¥ßO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚687654119149568 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42125499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692456084665597 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4801607926779 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚696017047225856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152399863277568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42125499 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42135476 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17748118 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9774513 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4801948476669 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 % _timestamp‚1.7843614161447575e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36056788 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692455727076347 trainer/global_step‚4549 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9774767 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696016688847360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36056788 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17747861 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4861696572416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691207741923328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36056786 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7973348 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157261650524160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4808946924032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚691207741923328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚3908.220272588 _step‚372 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691207758962688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚687654136188928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696017047225856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4809288263168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36056786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4861787246592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688847360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4808946924032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152399844665344ôE¯z;2¦- _wandbruntime‚3908  _runtime‚3908¾›ê„K§äM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691354982952960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4809959550976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78208962 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42144542 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9776860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7843614167565484e+09 _step‚373 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134551 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134551 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300897280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚687800646860800 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152433273423872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4862804485632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17751667 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959550976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36064418 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692603606391303 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157295985145344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4802618980501 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42144542 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696165300889600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36064420 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36064418 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300889600 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4810300897280 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4862711721472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36064420 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691354999992320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530503T„£ï&¨éM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300889600 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4810300897280 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4862711721472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36064420 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691354999992320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530503 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691354982952960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4809959550976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78208962 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42144542 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9776860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7843614167565484e+09 _step‚373 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134551 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134551 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300897280 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚687800646860800 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152433273423872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4862804485632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17751667 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959550976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36064418 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692603606391303 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157295985145344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4802618980501 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42144542 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696165300889600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36064420 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36064418 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664õ ÷€2©- _wandbruntime‚3908  _runtime‚3908Q…ƒÑª€ _step‚374 /loss/FRt2/detection_loss‚1.6203623191664616 /loss/FBt1/detection_loss‚0.6447835760457176 _runtime‚3908.834091606 1loss/SRqqt1/assignment_loss‚0.573700684606605 2loss/SRqqt2/assignment_loss‚0.5244374189309013 1loss/FBt2/assignment_loss‚0.03180917964541033 0loss/FRt1/assignment_loss‚1.8246750963314642 /loss/FRt1/detection_loss‚0.5640429165704439 1loss/SRqqt2/detection_loss‚0.6418140475783487 %loss/total_loss‚8.593177062891062 epoch‚11 $ _timestamp‚1.784361416759036e+09 /loss/FRt2/assignment_loss‚0.666720920085907 2loss/FBt1/assignment_loss‚0.007573079304449136 1loss/SRqqt1/detection_loss‚0.9585268329476931 /loss/FBt2/detection_loss‚0.5347309916776593 trainer/global_step‚4549‹ð¨ «… 2loss/FBt1/assignment_loss‚0.007573079304449136 1loss/SRqqt1/detection_loss‚0.9585268329476931 /loss/FBt2/detection_loss‚0.5347309916776593 trainer/global_step‚4549 _step‚374 /loss/FRt2/detection_loss‚1.6203623191664616 /loss/FBt1/detection_loss‚0.6447835760457176 _runtime‚3908.834091606 1loss/SRqqt1/assignment_loss‚0.573700684606605 2loss/SRqqt2/assignment_loss‚0.5244374189309013 1loss/FBt2/assignment_loss‚0.03180917964541033 0loss/FRt1/assignment_loss‚1.8246750963314642 /loss/FRt1/detection_loss‚0.5640429165704439 1loss/SRqqt2/detection_loss‚0.6418140475783487 %loss/total_loss‚8.593177062891062 epoch‚11 $ _timestamp‚1.784361416759036e+09 /loss/FRt2/assignment_loss‚0.666720920085907ö*•Ëw“¬‚(:ˆ ÎãìÒŒòź gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äí½¡ù­‚(:î ÎãìÒÿÖà¼# memory_percent‚97.0059871673584* proc.memory.percent‚16.741514205932617) disk./.usagePercent‚39.52591997433863( disk.nvme0c0n1.out‚12193.35302734375& disk.nvme0n1.out‚12193.35302734375 disk.drbd2165.in‚0 cpu‚26.181649415523765 disk.sda.out‚ 267.82421875 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚93.76416015625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 6039.08203125# disk.dm-0.out‚12193.35302734375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2742.9296875 disk.sda3.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125' proc.memory.availableMB‚ 490.5390625 network.recv‚ 1.063829e+06# disk.nvme0c0n1.in‚ 6039.08203125 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 267.82421875 network.sent‚ 7.102393e+06# disk./.usageGB‚588.964771270752! disk.nvme0n1.in‚ 6039.08203125 disk.drbd2165.out‚0 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 267.82421875„¾¬Åž®‚(:“ ÝãìÒ³Éຠgpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kÀ§ü¯‚(:ñ ÝãìÒ„†Å¼ disk.drbd2165.in‚0# disk.dm-0.out‚12197.94384765625 network.sent‚ 7.165271e+06) disk./.usagePercent‚39.52592560650437 disk.sda2.in‚0& disk.nvme0n1.out‚12197.94384765625 disk.sda1.in‚0( proc.memory.availableMB‚ 490.59765625 cpu‚26.180773734794453 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12197.94384765625 disk.sda3.out‚ 268.09765625" proc.memory.rssMB‚ 2742.9296875$ disk./.usageGB‚588.9648551940918 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 6094.37890625 disk.dm-1.in‚ 0.00390625# disk.nvme0c0n1.in‚ 6094.37890625 disk.sda1.out‚0* proc.memory.percent‚16.741514205932617 network.recv‚ 1.068544e+06 disk.rbd1.out‚93.84228515625! disk.nvme0n1.in‚ 6094.37890625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 268.09765625 disk.rbd1.in‚ 0.01953125 disk.dm-1.out‚ 268.09765625 disk.sda3.in‚ 0.00390625$ memory_percent‚97.00562953948975 proc.cpu.threads‚17 disk.rbd0.out‚0•†œ“°‚(:ˆ ìãìÒžàŒ» gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€«ü)ﱂ(:ä ìãìÒ˜Åõ¼ network.recv‚ 1.071842e+06 disk.dm-1.out‚ 268.31640625 disk.rbd1.out‚93.84228515625 network.sent‚ 7.170315e+06) disk./.usagePercent‚39.52591613422561 disk.rbd0.in‚0 disk.nvme0n1.in‚ 6094.3828125$ disk.nvme0n1.out‚12202.326171875 disk.sda3.in‚ 0.00390625! disk.dm-0.out‚12202.326171875& proc.memory.availableMB‚ 490.796875 disk.sda.out‚ 268.31640625 disk.drbd2165.out‚0 disk.dm-0.in‚ 6094.3828125 disk.rbd1.in‚ 0.01953125 disk.sda.in‚ 0.00390625 disk.sda1.in‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2742.9296875 cpu‚26.18040023840351" disk.nvme0c0n1.in‚ 6094.3828125& disk.nvme0c0n1.out‚12202.326171875 disk.sda3.out‚ 268.31640625 disk.dm-1.in‚ 0.00390625$ memory_percent‚97.00441360473633 disk.sda1.out‚0 disk.sda2.out‚0# disk./.usageGB‚588.964714050293 disk.rbd0.out‚0 disk.sda2.in‚0* proc.memory.percent‚16.741514205932617 proc.cpu.threads‚17Ø×2²- _wandbruntime‚3949  _runtime‚3949~°(ȳ ' lr-AdamW/pg2‚0.00038756988184857885 trainer/global_step‚4599 $ _timestamp‚1.784361456930813e+09 _runtime‚3949.007240496 _step‚375 ' lr-AdamW/pg1‚0.00038756988184857885®¢h/Í´Ç ' lr-AdamW/pg1‚0.00038756988184857885 ' lr-AdamW/pg2‚0.00038756988184857885 trainer/global_step‚4599 $ _timestamp‚1.784361456930813e+09 _runtime‚3949.007240496 _step‚375÷#Ω2µ- _wandbruntime‚3949  _runtime‚3949onݶÚO BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9877879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158984200224768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4912917633024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42579951 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚698569793404928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154071301203968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859642425856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36438386 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9878151 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚699832234746105 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36438388 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8060601 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158984301122560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703429435830784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17938477 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17938752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912999918592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚699831877156855 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435830784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚698569810444288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.½<¯c freed‚154071282591744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4859983764992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4852564963577 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚698569810444288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694979669782528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859642425856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42589928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360#€ å'·ßO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17938752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912999918592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚699831877156855 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435830784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚698569810444288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154071282591744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4859983764992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4852564963577 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚698569810444288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694979669782528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859642425856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42589928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9877879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158984200224768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4912917633024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42579951 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚698569793404928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154071301203968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859642425856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36438386 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9878151 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚699832234746105 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36438388 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8060601 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158984301122560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703429435830784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17938477 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4599ø¡:åó2¸- _wandbruntime‚3949  _runtime‚3949¢“.ë&¹åM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17942277 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36446018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4853576017411 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062343 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703578047873024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598994 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018632860160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035021 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4914024759808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695126180454400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17942560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154104711350272 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703578047873024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42589003 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018754722304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42589003 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882449 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695126163415040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655052800 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4599 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4860996399104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698717034434560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚3949.622528184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062346 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698717051473920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4913921509888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4853235467409 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36446020 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018632860160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035021 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4914024759808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695126180454400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17942560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154104711350272 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚703578047873024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42589003 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018754722304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42589003 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882449 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695126163415040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655052800 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4599 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4860996399104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698717034434560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚3949.622528184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062346 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698717051473920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4913921509888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4853235467409 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36446020 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17942277 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36446018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4853576017411 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062343 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703578047873024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598994 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/al¥9…×llocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚377ùÕM|¹2»- _wandbruntime‚3949  _runtime‚3949J$~ ¼ƒ 2loss/SRqqt1/assignment_loss‚0.5098451207389534 /loss/FRt1/detection_loss‚0.5614367122475347 /loss/FRt2/detection_loss‚1.4524083096774836 $loss/total_loss‚8.35788249526924 epoch‚11 % _timestamp‚1.7843614575495462e+09 0loss/FRt1/assignment_loss‚1.7950139006057824 1loss/SRqqt2/assignment_loss‚0.550766715920168 1loss/SRqqt2/detection_loss‚0.6369685103512266 /loss/FBt2/detection_loss‚0.5489508390099324 trainer/global_step‚4599 _runtime‚3949.624614823 2loss/FBt2/assignment_loss‚0.012262476816123852 _step‚378 0loss/FRt2/assignment_loss‚0.7393914027399547 3loss/FBt1/assignment_loss‚0.0001262983961962802 1loss/SRqqt1/detection_loss‚1.0027767109284145 /loss/FBt1/detection_loss‚0.5479354978374699÷ƒñ½ˆ 2loss/FBt2/assignment_loss‚0.012262476816123852 _step‚378 0loss/FRt2/assignment_loss‚0.7393914027399547 3loss/FBt1/assignment_loss‚0.0001262983961962802 1loss/SRqqt1/detection_loss‚1.0027767109284145 /loss/FBt1/detection_loss‚0.5479354978374699 2loss/SRqqt1/assignment_loss‚0.5098451207389534 /loss/FRt1/detection_loss‚0.5614367122475347 /loss/FRt2/detection_loss‚1.4524083096774836 $loss/total_loss‚8.35788249526924 epoch‚11 % _timestamp‚1.7843614575495462e+09 0loss/FRt1/assignment_loss‚1.7950139006057824 1loss/SRqqt2/assignment_loss‚0.550766715920168 1loss/SRqqt2/detection_loss‚0.6369685103512266 /loss/FBt2/detection_loss‚0.5489508390099324 trainer/global_step‚4599 _runtime‚3949.624614823úŽ¾Tp›¾‚(: ûãìÒû¢Æ» gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q§†²ñ¿‚(:æ ûãìÒ©Œ¾* disk./.usagePercent‚39.525917926278346 disk.sda.out‚ 268.5859375 disk.dm-0.in‚ 6094.3828125" disk.dm-0.out‚12208.0341796875 disk.rbd1.in‚ 0.01953125 network.recv‚ 1.07555e+06 disk.rbd0.out‚0% disk.nvme0n1.out‚12208.0341796875( proc.memory.availableMB‚ 490.26953125* proc.memory.percent‚16.741514205932617 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.out‚ 268.5859375 disk.nvme0n1.in‚ 6094.3828125 disk.sda3.out‚ 268.5859375 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625" disk.nvme0c0n1.in‚ 6094.3828125' disk.nvme0c0n1.out‚12208.0341796875# memory_percent‚97.0076322555542 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0" proc.memory.rssMB‚ 2742.9296875 cpu‚26.17983060593073 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 disk.sda2.in‚0$ disk./.usageGB‚588.9647407531738 disk.sda1.in‚0 disk.rbd1.out‚93.92041015625 disk.rbd0.in‚0 network.sent‚ 7.175394e+06Ò o“À‚(:ˆ ŠäìÒºëòº gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚237.593' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãêúÁòÁ‚(:ç ŠäìÒŸ¥¥¼ disk.dm-1.in‚ 0.00390625 disk.nvme0n1.in‚ 6094.3828125 disk.sda3.out‚ 268.9453125 disk.sda1.in‚0 disk.dm-1.out‚ 268.9453125 disk.rbd1.in‚ 0.01953125% disk.nvme0n1.out‚12211.0869140625 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2742.9296875 cpu‚26.178743091479838 network.recv‚ 1.080105e+06 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚12211.0869140625 disk.sda2.in‚0 disk.rbd1.out‚93.92041015625 disk.sda.out‚ 268.9453125 disk.sda1.out‚0 network.sent‚ 7.238321e+06 disk.dm-0.in‚ 6094.3828125 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625$ memory_percent‚97.00632095336914$ disk./.usageGB‚588.9647941589355" disk.dm-0.out‚12211.0869140625" disk.nvme0c0n1.in‚ 6094.3828125 disk.sda2.out‚0& proc.memory.availableMB‚ 490.484375* proc.memory.percent‚16.741514205932617* disk./.usagePercent‚39.525921510383824'˱PžÂ‚(:“ ™äìÒÏ去 gpu.0.gpu‚40.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø£y¤öÂ(:ë ™äìÒ‰½è¼ network.sent‚ 7.243302e+06! disk.nvme0n1.in‚ 6154.01953125 disk.sda.in‚ 0.00390625 disk.sda2.out‚0& proc.memory.availableMB‚ 490.734375 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.01953125 network.recv‚ 1.08362e+06) disk./.usagePercent‚39.52594455106191# disk.nvme0c0n1.in‚ 6154.01953125 cpu‚26.177694366815704 disk.sda.out‚ 269.19140625* proc.memory.percent‚16.741514205932617% disk.nvme0n1.out‚12216.2763671875 disk.sda3.out‚ 269.19140625 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚93.98291015625$ memory_percent‚97.00479507446289$ disk./.usageGB‚588.9651374816895 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.dm-0.in‚ 6154.01953125" disk.dm-0.out‚12216.2763671875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚12216.2763671875 disk.sda2.in‚0 disk.dm-1.out‚ 269.19140625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2742.9296875 proc.cpu.threads‚17]3ã2Ä- _wandbruntime‚3990  _runtime‚3990cn.ÉÅà _step‚379 ' lr-AdamW/pg1‚0.00038647784386786695 ' lr-AdamW/pg2‚0.00038647784386786695 trainer/global_step‚4649 % _timestamp‚1.7843614980048277e+09 _runtime‚3990.080159541+ÆÎÆÈ ' lr-AdamW/pg1‚0.00038647784386786695 ' lr-AdamW/pg2‚0.00038647784386786695 trainer/global_step‚4649 % _timestamp‚1.7843614980048277e+09 _runtime‚3990.080159541 _step‚379ûû*ª©2Ç- _wandbruntime‚3990  _runtime‚39902ÂDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚705931861925888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36819988 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182814208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9981520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43034403 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4902840900595 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9981249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36819988 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710842541192704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182814208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4964030790656 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160706751308800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚702305186336768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 % _timestamp‚1.7843614980059383e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36819986 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18129371 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _runtime‚3990.081430831 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155742739130368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710842541192704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43044380 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4910337927680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43044380 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155742720518144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710842541192704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43044380 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4910337927680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43044380 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155742720518144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚705931861925888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36819988 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182814208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9981520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43034403 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4902840900595 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9981249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36819988 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710842541192704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182814208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4964030790656üyÞæ×2Ê- _wandbruntime‚3990  _runtime‚3990bçvËËäM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18132944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706079102955520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702451697008640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43043455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36827618 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4903851954317 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4911691900928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702451714048000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43043455 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43053446 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710990794856448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9983621 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4965044772352 _runtime‚3990.694641344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706079085916160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710990794856448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QEDeviceStatsMonit£€zÚor.on_train_batch_end/inactive_split.large_pool.freed‚8149593 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4911691900928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843614986191528e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706079085916160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710990436470784 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706079102955520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4965143827968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707355548962957 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÛXÀÖï&ÌéM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43043455 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43053446 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710990794856448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9983621 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4965044772352 _runtime‚3990.694641344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706079085916160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710990794856448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8149593 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4911691900928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843614986191528e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706079085916160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710990436470784 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚706079102955520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4965143827968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707355548962957 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18132944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚706079102955520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702451697008640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43043455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36827618 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4903851954317 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4911691900928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702451714048000ý‚ŒÍ2Í- _wandbruntime‚3990  _runtime‚3990T á΂ /loss/FBt1/detection_loss‚0.9304637735566268 /loss/FBt2/detection_loss‚0.4835885594968836 0loss/FRt1/assignment_loss‚1.7142364465342033 1loss/FBt2/assignment_loss‚0.03612635780284788 -loss/FRt2/detection_loss‚1.62941688689681 1loss/SRqqt1/detection_loss‚0.8144860001991981 trainer/global_step‚4649 _runtime‚3990.696000519 1loss/SRqqt2/detection_loss‚0.6861792229102325 epoch‚11 0loss/FRt2/assignment_loss‚0.9207348045251125 2loss/SRqqt2/assignment_loss‚0.6226984224725851 3loss/FBt1/assignment_loss‚0.0018775111711525265 .loss/FRt1/detection_loss‚0.592888430519036 % _timestamp‚1.7843614986209288e+09 _step‚382 3loss/SRqqt1/assignment_loss‚0.48006749210538613 %loss/total_loss‚8.912763908190072C±È χ /loss/FBt1/detection_loss‚0.9304637735566268 /loss/FBt2/detection_loss‚0.4835885594968836 0loss/FRt1/assignment_loss‚1.7142364465342033 1loss/FBt2/assignment_loss‚0.03612635780284788 -loss/FRt2/detection_loss‚1.62941688689681 1loss/SRqqt1/detection_loss‚0.8144860001991981 trainer/global_step‚4649 _runtime‚3990.696000519 1loss/SRqqt2/detection_loss‚0.6861792229102325 epoch‚11 0loss/FRt2/assignment_loss‚0.9207348045251125 2loss/SRqqt2/assignment_loss‚0.6226984224725851 3loss/FBt1/assignment_loss‚0.0018775111711525265 .loss/FRt1/detection_loss‚0.592888430519036 % _timestamp‚1.7843614986209288e+09 _step‚382 3loss/SRqqt1/assignment_loss‚0.48006749210538613 %loss/total_loss‚8.912763908190072þçKÖeœÐ‚(:‘ ¨äìÒ÷¥¢º gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª®ÝïÑ‚(:ä ¨äìҦွ network.sent‚ 7.248441e+06 network.recv‚ 1.086993e+06 disk.rbd1.out‚93.98291015625$ memory_percent‚97.00784683227539$ disk./.usageGB‚588.9650993347168 disk.rbd1.in‚ 0.01953125! disk.nvme0n1.in‚ 6154.01953125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6154.01953125' disk.nvme0c0n1.out‚12219.4853515625# proc.memory.rssMB‚ 2742.94140625 cpu‚26.177526944933707 disk.rbd0.out‚0 disk.sda3.out‚ 269.46875 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.00390625& proc.memory.availableMB‚ 490.234375) disk./.usagePercent‚39.52594199098657 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.dm-0.out‚12219.4853515625 disk.rbd0.in‚0 disk.dm-1.out‚ 269.46875* proc.memory.percent‚16.741585731506348 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.sda.out‚ 269.46875 disk.dm-0.in‚ 6154.01953125% disk.nvme0n1.out‚12219.4853515625 disk.sda1.in‚0{Mið‘Ò‚(:† ·äìÒ½õÕº gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žêg¼ûÓ‚(:ð ·äìÒª¿˜¼) disk./.usagePercent‚39.52595325531807 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚94.12353515625 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 network.recv‚ 1.091278e+06 disk.sda1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6154.01953125 disk.sda3.out‚ 269.71484375# memory_percent‚97.0029354095459* proc.memory.percent‚16.741585731506348$ disk./.usageGB‚588.9652671813965# disk.dm-0.out‚12224.77099609375 disk.rbd1.in‚ 0.01953125 disk.sda.out‚ 269.71484375# proc.memory.rssMB‚ 2742.94140625 disk.sda1.out‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 491.0390625 network.sent‚ 7.311321e+06 disk.drbd2165.out‚0 disk.dm-0.in‚ 6154.01953125 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6154.01953125 disk.sda.in‚ 0.00390625 proc.cpu.threads‚17 disk.dm-1.out‚ 269.71484375( disk.nvme0c0n1.out‚12224.77099609375& disk.nvme0n1.out‚12224.77099609375 cpu‚26.176941189477503+¸ 2Ô- _wandbruntime‚4030  _runtime‚4030î¾èžÉÕà trainer/global_step‚4699 % _timestamp‚1.7843615387448084e+09 _runtime‚4030.820164355 _step‚383 ' lr-AdamW/pg1‚0.00038537495274881193 ' lr-AdamW/pg2‚0.00038537495274881193Fq¦±ÎÖÈ ' lr-AdamW/pg1‚0.00038537495274881193 ' lr-AdamW/pg2‚0.00038537495274881193 trainer/global_step‚4699 % _timestamp‚1.7843615387448084e+09 _runtime‚4030.820164355 _step‚383ÿ_Ï@2×- _wandbruntime‚4030  _runtime‚40303'Øâ'ØÜO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929797632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37201588 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4961374768640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4961374768640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚4030.821337931 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37201588 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718255288176128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚714584177317871 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709630736969728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4953797937393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37201586 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43488855 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157414158444544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚713293896368128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718255288176128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033429504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162429338294272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714584534907121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8235101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43498832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8235098 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚713293913407488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10085246 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157414177056768 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37201586 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4961033429504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚713293896368128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80700420 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162429449677824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5015272621056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929797632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37201588 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4961374768640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4961374768640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚4030.821337931 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37201588 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718255288176128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚714584177317871 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709630736969728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4953797937393 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37201586 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43488855 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157414158444544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚713293896368128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718255288176128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033429504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162429338294272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714584534907121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8235101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43498832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8235098 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚713293913407488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10085246 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157414177056768 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37201586 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872€ 04,2Ú- _wandbruntime‚4031  _runtime‚4031®¥}Në&ÛåM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43497907 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5016197423616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚162463784626688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714732056632827 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43507898 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4962387402752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚385 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709777230602240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157447587203072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784361539358014e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18323807 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4031.433535657 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718403541839872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8236843 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37209218 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10086964 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚713441137397760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37209220 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576³ Ñð&ÜêM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8236843 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37209218 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10086964 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚713441137397760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37209220 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43497907 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5016197423616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚162463784626688 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714732056632827 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43507898 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4962387402752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚385 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709777230602240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157447587203072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784361539358014e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18323807 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4031.433535657 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718403541839872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438öñ^62Ý- _wandbruntime‚4031  _runtime‚4031ß²ÞòÞ‚ 0loss/FRt2/assignment_loss‚0.6881976505303713 2loss/SRqqt2/assignment_loss‚0.4898311173371088 1loss/SRqqt1/detection_loss‚0.8688014960450564 _runtime‚4031.435728638 _step‚386 /loss/FBt1/detection_loss‚0.8386384340581196 %loss/total_loss‚8.855981466732556 0loss/FRt1/assignment_loss‚1.7963663676102173 2loss/SRqqt1/assignment_loss‚0.5618550681781496 /loss/FRt2/detection_loss‚1.4285234787934868 1loss/SRqqt2/detection_loss‚0.6466998403072259 /loss/FBt2/detection_loss‚0.7833069635089487 trainer/global_step‚4699 % _timestamp‚1.7843615393606899e+09 1loss/FBt1/assignment_loss‚0.16505194158153422 1loss/FBt2/assignment_loss‚0.03691762456167241 /loss/FRt1/detection_loss‚0.5517914842206665 epoch‚11†L ߇ /loss/FBt2/detection_loss‚0.7833069635089487 trainer/global_step‚4699 % _timestamp‚1.7843615393606899e+09 1loss/FBt1/assignment_loss‚0.16505194158153422 1loss/FBt2/assignment_loss‚0.03691762456167241 /loss/FRt1/detection_loss‚0.5517914842206665 epoch‚11 0loss/FRt2/assignment_loss‚0.6881976505303713 2loss/SRqqt2/assignment_loss‚0.4898311173371088 1loss/SRqqt1/detection_loss‚0.8688014960450564 _runtime‚4031.435728638 _step‚386 /loss/FBt1/detection_loss‚0.8386384340581196 %loss/total_loss‚8.855981466732556 0loss/FRt1/assignment_loss‚1.7963663676102173 2loss/SRqqt1/assignment_loss‚0.5618550681781496 /loss/FRt2/detection_loss‚1.4285234787934868 1loss/SRqqt2/detection_loss‚0.6466998403072259‚(ôÍ'“à‚(:ˆ ÆäìÒݬ²º gpu.0.gpu‚33.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AZl©ûá‚(:ð ÆäìÒ´ýŠ¼$ memory_percent‚97.00806140899658# proc.memory.rssMB‚ 2742.94140625 network.sent‚ 7.316349e+06 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚588.9654235839844 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6154.01953125! disk.nvme0n1.in‚ 6154.01953125 disk.sda2.in‚0 disk.rbd1.out‚94.12353515625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 269.95703125 disk.sda2.out‚0 disk.dm-1.out‚ 269.95703125 cpu‚26.17690456888038 network.recv‚ 1.09461e+06) disk./.usagePercent‚39.52596375162698 disk.dm-0.in‚ 6154.01953125# disk.dm-0.out‚12227.47216796875 disk.rbd0.out‚0( proc.memory.availableMB‚ 490.19921875* proc.memory.percent‚16.741585731506348 proc.cpu.threads‚17 disk.sda.out‚ 269.95703125 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12227.47216796875& disk.nvme0n1.out‚12227.47216796875 disk.sda.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625Šf 8“â‚(:ˆ ÕäìҙĻ gpu.0.gpu‚54.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œQV}òã‚(:ç ÕäìÒ­ã¼' proc.memory.availableMB‚ 490.5078125$ disk./.usageGB‚588.9655151367188 disk.sda3.out‚ 270.29296875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚94.18603515625 network.sent‚ 7.379238e+06 network.recv‚ 1.099317e+06! disk.dm-0.out‚12232.646484375 disk.rbd1.in‚ 0.01953125 disk.nvme0n1.in‚ 6166.7421875 disk.rbd0.in‚0* proc.memory.percent‚16.741585731506348 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚12232.646484375 disk.dm-0.in‚ 6166.7421875" disk.nvme0c0n1.in‚ 6166.7421875$ memory_percent‚97.00617790222168# proc.memory.rssMB‚ 2742.94140625$ disk.nvme0n1.out‚12232.646484375 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625( disk./.usagePercent‚39.5259698958078 disk.sda.out‚ 270.29296875 disk.dm-1.out‚ 270.29296875 cpu‚26.176428038748742û²œíä‚(:â ääìÒ‡Ÿç¼ disk.rbd0.in‚0$ disk.nvme0n1.out‚12237.521484375$ disk./.usageGB‚588.9654350280762 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 270.49609375 disk.drbd2165.in‚0! disk.dm-0.out‚12237.521484375 disk.sda.out‚ 270.49609375 disk.nvme0c0n1.in‚ 6222.78125& disk.nvme0c0n1.out‚12237.521484375 disk.nvme0n1.in‚ 6222.78125 proc.cpu.threads‚17 network.recv‚ 1.102358e+06* disk./.usagePercent‚39.525964519649584 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 270.49609375 disk.rbd1.out‚94.18603515625 disk.sda2.out‚0$ memory_percent‚97.00970649719238* proc.memory.percent‚16.741585731506348 cpu‚26.17559718612286 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 7.384361e+06 disk.dm-1.in‚ 0.00390625' proc.memory.availableMB‚ 489.9296875# proc.memory.rssMB‚ 2742.94140625 disk.dm-0.in‚ 6222.78125 disk.rbd1.in‚ 0.01953125#ö2¸å‚(:ˆ ääìÒ‹™õ¼ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.P¬$Û0.powerWatts‚260.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åHÙ2æ- _wandbruntime‚4071  _runtime‚4071©,¶âÉçà _step‚387 ' lr-AdamW/pg1‚0.00038426127932804935 ' lr-AdamW/pg2‚0.00038426127932804935 trainer/global_step‚4749 % _timestamp‚1.7843615797228096e+09 _runtime‚4071.798115138pUNÎèÈ ' lr-AdamW/pg1‚0.00038426127932804935 ' lr-AdamW/pg2‚0.00038426127932804935 trainer/global_step‚4749 % _timestamp‚1.7843615797228096e+09 _runtime‚4071.798115138 _step‚387ƒä‰b2é- _wandbruntime‚4071  _runtime‚4071Ãëdá'êÛO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37583186 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676781056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728931328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18510398 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720655947849728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43943307 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5011728931328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5066289047552 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚725668035159552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37583188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10188050 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚716956253523968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398379 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5066375527424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚4071.801134548 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953284 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943307 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37583186 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18510666 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424301 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035159552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720655964889088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚720655964889088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164151990510592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚721960684987629 trainer/global_step‚4749 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496oÅmæ'ëàO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚388 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37583188 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720655947849728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚725667676781056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8322351 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5012070270464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5012070270464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526493 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676781056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728931328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18510398 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720655947849728 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43943307 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5011728931328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5066289047552 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚725668035159552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37583188 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10188050 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚716956253523968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398379 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5066375527424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚4071.801134548 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953284 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943307 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37583186 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18510666 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424301 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035159552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720655964889088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚720655964889088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164151990510592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚721960684987629 trainer/global_step‚4749 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37583186 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5012741558272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720803205918720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚ 4072.42129595 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10190072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717102781235200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4749 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928133 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164186331988480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚725816288823296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43952359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18514433 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37590818 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10190337 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543177 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741558272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚389 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43962350 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43952359 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478135 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8324096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717102764195840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37590820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37590818 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843615803457122e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚725815930437632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5067399623168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013082904576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164186443364864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18514165 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722107849123973 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5013082904576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5067306859008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543177 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741558272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚389 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43962350 SEDeviceStatsMonitor.on_train_Ú „—ð batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43952359 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478135 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8324096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717102764195840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37590820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37590818 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843615803457122e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚725815930437632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5067399623168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013082904576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164186443364864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18514165 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722107849123973 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5013082904576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5067306859008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5012741558272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720803205918720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚ 4072.42129595 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10190072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717102781235200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4749 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928133 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164186331988480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚725816288823296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43952359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18514433…˜Á2ï- _wandbruntime‚4072  _runtime‚4072­0µ¼ðÿ 2loss/FBt1/assignment_loss‚0.013048504308324027 2loss/FBt2/assignment_loss‚0.018045866257475198 %loss/total_loss‚8.610795094025981 epoch‚11 0loss/FRt2/assignment_loss‚0.5928713269624166 1loss/SRqqt2/assignment_loss‚0.544199321976677 0loss/SRqqt2/detection_loss‚0.662790382213531 /loss/FBt1/detection_loss‚0.7423911111429333 /loss/FBt2/detection_loss‚0.5521554897261214 trainer/global_step‚4749 % _timestamp‚1.7843615803473253e+09 1loss/SRqqt1/assignment_loss‚0.651536903836601 /loss/FRt1/detection_loss‚0.5392260524664447 _runtime‚4072.422396467 _step‚390 0loss/FRt1/assignment_loss‚1.7267280808882788 .loss/FRt2/detection_loss‚1.536687107479417 0loss/SRqqt1/detection_loss‚1.031114946767761ò*«ä ñ„ 2loss/FBt1/assignment_loss‚0.013048504308324027 2loss/FBt2/assignment_loss‚0.018045866257475198 %loss/total_loss‚8.610795094025981 epoch‚11 0loss/FRt2/assignment_loss‚0.5928713269624166 1loss/SRqqt2/assignment_loss‚0.544199321976677 0loss/SRqqt2/detection_loss‚0.662790382213531 /loss/FBt1/detection_loss‚0.7423911111429333 /loss/FBt2/detection_loss‚0.5521554897261214 trainer/global_step‚4749 % _timestamp‚1.7843615803473253e+09 1loss/SRqqt1/assignment_loss‚0.651536903836601 /loss/FRt1/detection_loss‚0.5392260524664447 _runtime‚4072.422396467 _step‚390 0loss/FRt1/assignment_loss‚1.7267280808882788 .loss/FRt2/detection_loss‚1.536687107479417 0loss/SRqqt1/detection_loss‚1.031114946767761†v}D™“ò‚(:ˆ óäìÒ× gpu.0.gpu‚44.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚144.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WW{Ãòó‚(:ç óäìÒ²íè» network.recv‚ 1.104832e+06 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 6222.78125 disk.nvme0n1.in‚ 6222.78125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 270.70703125 disk.sda.out‚ 270.70703125# disk.dm-0.out‚12244.38037109375& proc.memory.availableMB‚ 489.703125 cpu‚26.175394885242234 disk.sda.in‚ 0.00390625 disk.sda1.out‚0 network.sent‚ 7.389165e+06$ disk./.usageGB‚588.9653549194336 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.741585731506348$ memory_percent‚97.01108932495117 disk.rbd1.in‚ 0.01953125& disk.nvme0n1.out‚12244.38037109375 disk.sda1.in‚0 disk.sda3.out‚ 270.70703125 disk.dm-0.in‚ 6222.78125) disk./.usagePercent‚39.52595914349136 disk.rbd1.out‚94.26416015625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12244.38037109375 disk.sda3.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625# proc.memory.rssMB‚ 2742.94140625\»Ô“ô‚(:ˆ ‚åìÒÏ­ôº gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ëðçõ‚(:Ü ‚åìÒƒÁ¼ disk.sda.out‚ 271.0546875 disk.sda1.out‚0 disk.dm-0.in‚ 6222.78125 network.sent‚ 7.45238e+06 disk.sda.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0! disk.dm-0.out‚12247.318359375 disk.nvme0n1.in‚ 6222.78125' proc.memory.availableMB‚ 490.0078125$ disk./.usageGB‚588.9654312133789 disk.sda2.in‚0 disk.drbd2165.in‚0$ memory_percent‚97.00922966003418# proc.memory.rssMB‚ 2742.94140625 disk.sda3.in‚ 0.00390625 disk.dm-1.out‚ 271.0546875 disk.drbd2165.out‚0 network.recv‚ 1.109896e+06 disk.sda3.out‚ 271.0546875$ disk.nvme0n1.out‚12247.318359375 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚94.26416015625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚12247.318359375 cpu‚26.1760821842615) disk./.usagePercent‚39.52596426364205 disk.rbd1.in‚ 0.01953125 disk.nvme0c0n1.in‚ 6222.78125* proc.memory.percent‚16.741585731506348 proc.cpu.threads‚17#õžö‚(:“ ‘åìÒªö˜» gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚243.212' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.642399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ôÏïí÷‚(:â ‘åìÒ’ºò¼ disk.rbd0.in‚0# proc.memory.rssMB‚ 2742.94140625 disk.nvme0c0n1.in‚ 6222.78125 disk.dm-1.out‚ 271.296875$ memory_percent‚97.00801372528076( proc.memory.availableMB‚ 490.20703125 disk.dm-0.in‚ 6222.78125 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 271.296875 disk.sda2.out‚0 disk.rbd1.in‚ 0.01953125( disk.nvme0c0n1.out‚12251.99072265625 disk.sda.out‚ 271.296875 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.00390625 proc.cpu.threads‚17 network.recv‚ 1.113242e+06 disk.sda.in‚ 0.00390625 disk.sda1.out‚0 cpu‚26.177010403924843 disk.rbd1.out‚94.32666015625 disk.sda2.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚12251.99072265625 network.sent‚ 7.45735e+06$ disk./.usageGB‚588.9655609130859 disk.nvme0n1.in‚ 6222.78125& disk.nvme0n1.out‚12251.99072265625 disk.sda1.in‚0* proc.memory.percent‚16.741585731506348) disk./.usagePercent‚39.52597296789821 disk.rbd0.out‚0©§Cõ2ø- _wandbruntime‚4112  _runtime‚4112¬'ÜFÉùà ' lr-AdamW/pg1‚0.00038313689513474206 ' lr-AdamW/pg2‚0.00038313689513474206 trainer/global_step‚4799 % _timestamp‚1.7843616203548083e+09 _runtime‚4112.430189579 _step‚391G¯üdÎúÈ trainer/global_step‚4799 % _timestamp‚1.7843616203548083e+09 _runtime‚4112.430189579 _step‚391 ' lr-AdamW/pg1‚0.00038313689513474206 ' lr-AdamW/pg2‚0.00038313689513474206‡ÝÐÚ¿2û- _wandbruntime‚4112  _runtime‚4112–‚Óà'üÚO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚733080782142976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10290673 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚733080423764480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚728018016370688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚733080782142976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728018016370688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5062424433152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚729336835068137 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5054690361319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44397759 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423764480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚728017999331328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729336477478887 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚4799 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362545 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44407736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37964786 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165874337800704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚724281804156928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82372524 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚392å;úýßO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160757052909568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5062765772288 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚724281804156928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82372524 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚733080782142976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10290673 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚733080423764480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚728018016370688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚733080782142976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728018016370688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5062424433152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚729336835068137 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5054690361319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44397759 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423764480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚728017999331328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729336477478887 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚4799 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362545 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44407736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37964786 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165874337800704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37972420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚4113.045756274 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚728165257400320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18703829 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165908895042560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406811 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5063437060096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18704104 trainer/global_step‚4799 % _timestamp‚1.7843616209702015e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5063778406400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5118413374464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165908781569024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724428314828800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406811 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10292758 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚729483999204481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8411346 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724428297789440 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160790463055872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37972418 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728165240360960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437060096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚733229035806720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160790481668096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37972418 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728165257400320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0çž ñ&€ëM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8411346 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724428297789440 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160790463055872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37972418 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728165240360960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437060096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚733229035806720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160790481668096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37972418 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728165257400320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37972420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚4113.045756274 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚728165257400320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18703829 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165908895042560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406811 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5063437060096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18704104 trainer/global_step‚4799 % _timestamp‚1.7843616209702015e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5063778406400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5118413374464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165908781569024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724428314828800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406811 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10292758 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚729483999204481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0‰á“ò2- _wandbruntime‚4113  _runtime‚4113B¾÷ ‚„ 2loss/SRqqt1/assignment_loss‚0.5429875078513093 3loss/SRqqt2/assignment_loss‚0.46923847976050054 2loss/FBt1/assignment_loss‚0.000315948374436473 %loss/total_loss‚8.306234485926073 epoch‚11 _step‚394 0loss/FRt1/assignment_loss‚1.7865969756726663 0loss/FBt2/detection_loss‚0.43415428193197036 trainer/global_step‚4799 _runtime‚4113.047191163 2loss/FBt2/assignment_loss‚0.006408422869626944 /loss/FRt1/detection_loss‚0.5670288736307175 1loss/SRqqt1/detection_loss‚0.8814622489518176 0loss/FRt2/assignment_loss‚0.7841986645835106 /loss/FRt2/detection_loss‚1.5200492977503548 1loss/SRqqt2/detection_loss‚0.6427600930627559 .loss/FBt1/detection_loss‚0.671033691486408 $ _timestamp‚1.784361620972108e+09ê¸Pƒ‰ 2loss/FBt2/assignment_loss‚0.006408422869626944 /loss/FRt1/detection_loss‚0.5670288736307175 1loss/SRqqt1/detection_loss‚0.8814622489518176 0loss/FRt2/assignment_loss‚0.7841986645835106 /loss/FRt2/detection_loss‚1.5200492977503548 1loss/SRqqt2/detection_loss‚0.6427600930627559 .loss/FBt1/detection_loss‚0.671033691486408 $ _timestamp‚1.784361620972108e+09 2loss/SRqqt1/assignment_loss‚0.5429875078513093 3loss/SRqqt2/assignment_loss‚0.46923847976050054 2loss/FBt1/assignment_loss‚0.000315948374436473 %loss/total_loss‚8.306234485926073 epoch‚11 _step‚394 0loss/FRt1/assignment_loss‚1.7865969756726663 0loss/FBt2/detection_loss‚0.43415428193197036 trainer/global_step‚4799 _runtime‚4113.047191163Šõd:"“„‚(:ˆ  åìÒ³…º gpu.0.gpu‚81.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¿æGð…‚(:å  åìÒèÐǼ disk.dm-0.in‚ 6275.40625 disk.sda2.out‚0 disk.nvme0n1.in‚ 6275.40625 disk.sda.in‚ 0.00390625# proc.memory.rssMB‚ 2746.17578125$ disk./.usageGB‚588.9654884338379 disk.drbd2165.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚12254.84716796875& disk.nvme0n1.out‚12254.84716796875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 271.68359375 disk.dm-1.in‚ 0.00390625 network.sent‚ 7.462379e+06 network.recv‚ 1.116474e+06) disk./.usagePercent‚39.52596810375506 disk.sda1.in‚0( disk.nvme0c0n1.out‚12254.84716796875$ memory_percent‚97.02949523925781* proc.memory.percent‚16.761326789855957 disk.sda1.out‚0 disk.rbd1.in‚ 0.01953125 disk.sda.out‚ 271.68359375 cpu‚26.187400864454226 disk.drbd2165.in‚0 disk.rbd1.out‚94.32666015625 proc.cpu.threads‚17$ proc.memory.availableMB‚486.6875 disk.sda2.in‚0 disk.dm-1.out‚ 271.68359375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 6275.40625•|«2†- _wandbruntime‚4130  _runtime‚4130÷£Àѹ‡ž{ + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8132851123809814 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1861516386270523 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 )loss/val_total_loss‚8.404902113924512 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8903250694274902 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9015495777130127 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7641605138778687 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26949915289878845 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.762413740158081 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.853943943977356 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5842040181159973 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17869703471660614 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47834286093711853 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7663449645042419 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03960498422384262 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14791791141033173 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03459874913096428 _runtime‚ 4130.95264094 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8874607086181641 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 jet/accuracy_3_of_5‚1 $ _timestamp‚1.784361638876744e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980957508087158 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.042501140385866165 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4972287714481354 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7358427047729492 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1994127780199051 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16553886234760284 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17979001998901367 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1869504749774933 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04618353769183159 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7067857384681702 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0378948412835598 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7676792144775391 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3217553496360779 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17757070064544678 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8431956171989441 *jet/accuracy_1_of_2‚0.8056628108024597 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_3‚0.9314910173416138 *jet/accuracy_2_of_4‚0.8054164052009583 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19882211089134216 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7346680164337158 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5710033178329468 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02022814191877842 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5845829844474792 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7394911050796509 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6395392417907715 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 'particle/f_score‚0.6462340354919434 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6442031264305115 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6663289070129395 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04285500571131706 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9744162559509277 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.894255518913269 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11595369875431061 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24142690002918243 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.786989688873291 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3922420144081116 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6572014689445496 6validation_average_jet_accuracy‚0.3688877820968628 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13111841678619385 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.872430145740509 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8898337483406067 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6343933343887329 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3142077624797821 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7634140849113464 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚•ûâë!0.19704173505306244 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03960498422384262 particle/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 _step‚395 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6617621779441833 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9934728741645813 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6770925521850586 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1998780369758606 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.265625 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7154141068458557 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39313653111457825 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 (particle/accuracy‚0.8498923182487488 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8990691304206848 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4560185670852661 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7648976445198059 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6512802839279175 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.052424363791942596 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7287911772727966 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_5_of_5‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.983397901058197 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994524359703064 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7662968635559082 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14270471036434174 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.193841814994812 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 epoch‚11 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7941420674324036 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13221658766269684 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22312262654304504 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9077075123786926 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15469832718372345 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04326234385371208 /particle/accuracy_4_of_4‚0.9522042274475098 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9500647783279419 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47012418508529663 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01628045178949833 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25379252433776855 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20252461731433868 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8150360584259033 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948445558547974 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955971837043762 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7885503172874451 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933582544326782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_3_of_3‚0.07612902671098709 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08416666835546494 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/sensitivity‚0.6384871602058411 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1861516386270523 particle/accuracy_1_of_1‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6844136118888855 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16388408839702606 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5655222535133362 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6445351243019104 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.484375 +jet/accuracy_3_of_4‚0.10002080351114273 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7101566195487976 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_2_of_2‚0.15035882592201233 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02776365354657173 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6706303954124451 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3070429861545563 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4290663003921509 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6638826727867126 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04218599945306778 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6677284240722656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6663289070129395 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5291752219200134 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6040876507759094 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17976590991020203 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9851412177085876 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2013724148273468 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897873401641846 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6627390384674072 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21540601551532745 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8431956171989441 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 trainer/global_step‚4799 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971871376037598 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6171875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45856717228889465 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7113844752311707 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44570377469062805‚À@©=ˆ£{ :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14270471036434174 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.193841814994812 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 epoch‚11 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7941420674324036 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13221658766269684 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22312262654304504 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9077075123786926 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15469832718372345 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04326234385371208 /particle/accuracy_4_of_4‚0.9522042274475098 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9500647783279419 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47012418508529663 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01628045178949833 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25379252433776855 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20252461731433868 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8150360584259033 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948445558547974 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955971837043762 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7885503172874451 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9933582544326782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_3_of_3‚0.07612902671098709 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08416666835546494 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/sensitivity‚0.6384871602058411 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1861516386270523 particle/accuracy_1_of_1‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6844136118888855 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16388408839702606 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5655222535133362 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6445351243019104 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.484375 +jet/accuracy_3_of_4‚0.10002080351114273 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7101566195487976 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_2_of_2‚0.15035882592201233 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02776365354657173 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6706303954124451 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3070429861545563 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4290663003921509 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6638826727867126 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04218599945306778 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6677284240722656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6663289070129395 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5291752219200134 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6040876507759094 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17976590991020203 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9851412177085876 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2013724148273468 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897873401641846 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6627390384674072 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21540601551532745 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8431956171989441 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 trainer/global_step‚4799 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971871376037598 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6171875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45856717228889465 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7113844752311707 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44570377469062805 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8132851123809814 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1861516386270523 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 )loss/val_total_loss‚8.404902113924512 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8903250694274902 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9015495777130127 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7641605138778687 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26949915289878845 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.762413740158081 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.853943943977356 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5842040181159973 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17869703471660614 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47834286093711853 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7663449645042419 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03960498422384262 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14791791141033173 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03459874913096428 _runtime‚ 4130.95264094 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8874607086181641 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 jet/accuracy_3_of_5‚1 $ _timestamp‚1.784361638876744e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980957508087158 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.042501140385866165 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4972287714481354 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7358427047729492 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1994127780199051 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16553886234760284 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17979001998901367 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1869504749774933 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04618353769183159 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7067857384681702 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0378948412835598 /particle/accuracy_3_of_4‚0.9968254566192627 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7676792144775391 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3217553496360779 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17757070064544678 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8431956171989441 *jet/accuracy_1_of_2‚0.8056628108024597 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_3‚0.9314910173416138 *jet/accuracy_2_of_4‚0.8054164052009583 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19882211089134216 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7346680164337158 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5710033178329468 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02022814191877842 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5845829844474792 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7394911050796509 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6395392417907715 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 'particle/f_score‚0.6462340354919434 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6442031264305115 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6663289070129395 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04285500571131706 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9744162559509277 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.894255518913269 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11595369875431061 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24142690002918243 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.786989688873291 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3922420144081116 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6572014689445496 6validation_average_jet_accuracy‚0.3688877820968628 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_2_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13111841678619385 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.872430145740509 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8898337483406067 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6343933343887329 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3142077624797821 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7634140849113464 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19704173505306244 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03960498422384262 particle/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 _step‚395 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6617621779441833 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9934728741645813 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6770925521850586 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1998780369758606 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.265625 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7154141068458557 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39313653111457825 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 (particle/accuracy‚0.8498923182487488 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8990691304206848 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4560185670852661 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7648976445198059 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6512802839279175 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.052424363791942596 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7287911772727966 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_5_of_5‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.983397901058197 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994524359703064 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7662968635559082 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172‹­¦ZËž‰‚(:“ ¯åìÒ¸À„» gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡^ÚwôŠ‚(:é ¯åìÒÚ…–½ disk.sda2.in‚0 disk.dm-0.in‚ 6275.40625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 271.95703125 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 271.95703125$ disk./.usageGB‚588.9655113220215 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 271.95703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 6275.40625' proc.memory.availableMB‚ 484.1796875# proc.memory.rssMB‚ 2748.80859375 network.recv‚ 1.120765e+06* disk./.usagePercent‚39.525969639800266 disk.rbd1.in‚ 0.01953125 cpu‚26.193081245857993 network.sent‚ 7.525261e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚12260.45849609375 disk.rbd0.out‚0& disk.nvme0n1.out‚12260.45849609375 disk.sda3.in‚ 0.00390625( disk.nvme0c0n1.out‚12260.45849609375$ memory_percent‚97.04480171203613 proc.cpu.threads‚17 disk.rbd1.out‚94.40478515625 disk.nvme0n1.in‚ 6275.40625* proc.memory.percent‚16.777396202087402êæeý“‹‚(:ˆ ¾åìÒÜö» gpu.0.gpu‚65.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚155.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.1732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ˆ/}茂(:Ý ¾åìÒ̳ö¼% disk.nvme0n1.out‚12264.9521484375& proc.memory.availableMB‚ 484.234375 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚94.40478515625 disk.rbd0.out‚0 disk.sda.out‚ 272.140625 network.sent‚ 7.582614e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 6275.40625' disk.nvme0c0n1.out‚12264.9521484375# proc.memory.rssMB‚ 2748.80859375 disk.nvme0n1.in‚ 6275.40625 disk.sda3.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 6275.40625" disk.dm-0.out‚12264.9521484375 network.recv‚ 1.124804e+06$ disk./.usageGB‚588.9654502868652 disk.sda1.out‚0 disk.sda2.in‚0) disk./.usagePercent‚39.52596554367972 disk.dm-1.out‚ 272.140625$ memory_percent‚97.04446792602539 cpu‚26.19382708983266 disk.sda3.out‚ 272.140625* proc.memory.percent‚16.777396202087402 proc.cpu.threads‚17ê±ùŠó‚(:è ÍåìÒ§™Ê¼ network.recv‚ 1.127795e+06$ disk./.usageGB‚588.9654426574707 proc.cpu.threads‚17 disk.dm-1.out‚ 272.43359375 disk.sda.out‚ 272.43359375 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0( proc.memory.availableMB‚ 484.09765625 disk.sda3.out‚ 272.43359375 disk.sda1.in‚0# proc.memory.rssMB‚ 2748.80859375) disk./.usagePercent‚39.52596503166465 disk.nvme0n1.in‚ 6275.40625 disk.sda.in‚ 0.00390625 network.sent‚ 7.587515e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12271.40185546875$ memory_percent‚97.04530239105225* proc.memory.percent‚16.777396202087402 cpu‚26.19468820094873 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 disk.nvme0c0n1.in‚ 6275.40625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6275.40625 disk.rbd1.out‚94.46728515625 disk.rbd0.in‚0& disk.nvme0n1.out‚12271.40185546875# disk.dm-0.out‚12271.40185546875‘‰ô‚(:“ ÍåìÒ”Ó›½ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ÿP2- _wandbruntime‚4170  _runtime‚4170Óc¯Éà _step‚396 ' lr-AdamW/pg1‚0.00038200187238598575 ' lr-AdamW/pg2‚0.00038200187238598575 trainer/global_step‚4849 % _timestamp‚1.7843616788137333e+09 _runtime‚4170.889104534ü• Î‘È _runtime‚4170.889104534 _step‚396 ' lr-AdamW/pg1‚0.00038200187238598575 ' lr-AdamW/pg2‚0.00038200187238598575 trainer/global_step‚4849 % _timestamp‚1.7843616788137333e+09ŒØ×r¶2’- _wandbruntime‚4170  _runtime‚4170Q@[Ëx “ÛO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44922533 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470603776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44922533 trainer/global_step‚4849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5186731892736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18926036 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚738639010498704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5123367123346 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5130853140480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167631040222208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738639368087954 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733515983925248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚737309617463296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10426411 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742440470603776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38467052 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚737309634502656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742440828982272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136¤$!µi KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162444402935808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38467054 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5186655898624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚733516000964608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8499368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389585 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742440828982272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5130853140480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167631134828544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38467054 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚737309617463296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5123026573456 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44932510 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5131194479616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5131194479616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10426665 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4170.890305241 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470603776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44922533 trainer/global_step‚4849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5186731892736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18926036 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚738639010498704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5123367123346 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5130853140480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167631040222208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738639368087954 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733515983925248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚737309617463296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10426411 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742440470603776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38467052 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚737309634502656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742440828982272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162444402935808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38467054 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5186655898624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚733516000964608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8499368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389585 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742440828982272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5130853140480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167631134828544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38467054 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚737309617463296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5123026573456 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44932510 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5131194479616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5131194479616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10426665 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4170.890305241 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742589082646016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737456875532288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737456858492928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5187678362624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5124037627178 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18929602 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44941576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚737456875532288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5132207113728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167665596529664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18929863 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _runtime‚ 4171.50309139 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742588724260352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚733662511636480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.784361679427531e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚737456858492928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38474684 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742589082646016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0à#Hr—éM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚733662511636480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.784361679427531e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚737456858492928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38474684 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742589082646016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10428489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5131865767424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742589082646016 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73FÀl¥}7456875532288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737456858492928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5187678362624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5124037627178 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18929602 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44941576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚737456875532288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5132207113728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167665596529664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18929863 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _runtime‚ 4171.50309139 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742588724260352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438Žqé4À2˜- _wandbruntime‚4171  _runtime‚4171Vzº™‚ /loss/FRt1/detection_loss‚0.5611374540798828 /loss/FRt2/detection_loss‚1.4638334204022567 1loss/SRqqt2/detection_loss‚0.6732633104882385 /loss/FBt2/detection_loss‚0.5037285469050131 _step‚399 0loss/FRt1/assignment_loss‚1.7642256151354698 0loss/FRt2/assignment_loss‚0.5709165051216032 2loss/FBt2/assignment_loss‚0.005323278933721651 /loss/FBt1/detection_loss‚0.8262078445244195 $loss/total_loss‚8.31864925984264 _runtime‚4171.505102326 2loss/SRqqt1/assignment_loss‚0.4847102037109653 1loss/SRqqt2/assignment_loss‚0.515716671516973 1loss/SRqqt1/detection_loss‚0.9415014447243167 trainer/global_step‚4849 % _timestamp‚1.7843616794300292e+09 2loss/FBt1/assignment_loss‚0.008084964299780888 epoch‚12W® š‡ /loss/FBt1/detection_loss‚0.8262078445244195 $loss/total_loss‚8.31864925984264 _runtime‚4171.505102326 2loss/SRqqt1/assignment_loss‚0.4847102037109653 1loss/SRqqt2/assignment_loss‚0.515716671516973 1loss/SRqqt1/detection_loss‚0.9415014447243167 trainer/global_step‚4849 % _timestamp‚1.7843616794300292e+09 2loss/FBt1/assignment_loss‚0.008084964299780888 epoch‚12 /loss/FRt1/detection_loss‚0.5611374540798828 /loss/FRt2/detection_loss‚1.4638334204022567 1loss/SRqqt2/detection_loss‚0.6732633104882385 /loss/FBt2/detection_loss‚0.5037285469050131 _step‚399 0loss/FRt1/assignment_loss‚1.7642256151354698 0loss/FRt2/assignment_loss‚0.5709165051216032 2loss/FBt2/assignment_loss‚0.005323278933721651]¡èˆ“›‚(:ˆ ÜåìÒŒ‹—º gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù=³÷œ‚(:ì ÜåìÒϤ¼ cpu‚26.195130309975152 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 272.83203125 disk.sda1.out‚0 network.recv‚ 1.132416e+06 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6338.16796875 disk.sda.out‚ 272.83203125( proc.memory.availableMB‚ 483.91015625$ memory_percent‚97.04644680023193 network.sent‚ 7.650506e+06 disk.sda2.in‚0 disk.dm-1.out‚ 272.83203125 disk.rbd1.in‚ 0.01953125 disk.rbd0.out‚0$ disk./.usageGB‚588.9655570983887 disk.drbd2165.in‚0 disk.dm-0.in‚ 6338.16796875 disk.rbd1.out‚94.46728515625# disk.nvme0c0n1.in‚ 6338.16796875* proc.memory.percent‚16.777777671813965 disk.sda2.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2748.87109375) disk./.usagePercent‚39.52597271189068 disk.sda.in‚ 0.00390625$ disk.nvme0n1.out‚12275.888671875 disk.sda3.in‚ 0.00390625! disk.dm-0.out‚12275.888671875& disk.nvme0c0n1.out‚12275.888671875ÎŒ-9“‚(:ˆ ëåìÒ¨õ”» gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åE‹ñž‚(:æ ëåìÒ¡Øå¼* proc.memory.percent‚16.777777671813965$ disk./.usageGB‚588.9654159545898 disk.sda2.in‚0 disk.dm-1.out‚ 273.046875 cpu‚26.19590667853213) disk./.usagePercent‚39.52596323961191 disk.rbd1.in‚ 0.01953125& disk.nvme0n1.out‚12281.60009765625 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 273.046875 disk.sda1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.04666137695312 network.sent‚ 7.655589e+06 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6338.16796875 disk.sda3.in‚ 0.00390625# disk.nvme0c0n1.in‚ 6338.16796875 disk.sda.in‚ 0.00390625 disk.sda.out‚ 273.046875 disk.rbd1.out‚94.60791015625( disk.nvme0c0n1.out‚12281.60009765625# proc.memory.availableMB‚483.875 proc.cpu.threads‚17 network.recv‚ 1.136143e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 6338.16796875# disk.dm-0.out‚12281.60009765625 disk.dm-1.in‚ 0.00390625# proc.memory.rssMB‚ 2748.87109375Ó™A2Ÿ- _wandbruntime‚4211  _runtime‚4211õ ÁÇ Á _runtime‚4211.211103418 _step‚400 ' lr-AdamW/pg1‚0.00038085628398217046 ' lr-AdamW/pg2‚0.00038085628398217046 trainer/global_step‚4899 # _timestamp‚1.78436171913578e+09‡§"¢Ì¡Æ _runtime‚4211.211103418 _step‚400 ' lr-AdamW/pg1‚0.00038085628398217046 ' lr-AdamW/pg2‚0.00038085628398217046 trainer/global_step‚4899 # _timestamp‚1.78436171913578e+09—úM2¢- _wandbruntime‚4211  _runtime‚4211O ?2á'£ÛO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169354108166144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚746015160579212 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5238267303936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5238187115520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38848652 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740841534558208 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚740841517518848 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744671685984256 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10531138 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169354009365504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38848654 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚749853575965696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181548642304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45386962 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181889981440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4899 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8586618 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45376985 _runtime‚4211.212340834 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38848654 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8586621 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5181889981440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217587200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843617191368308e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225637 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚744671668944896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164115840862208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181548642304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5173983610254 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45386962 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217587200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843617191368308e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225637 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚744671668944896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164115840862208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181548642304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5173983610254 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45386962 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169354108166144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚746015160579212 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5238267303936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5238187115520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38848652 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740841534558208 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚740841517518848 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744671685984256 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10531138 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169354009365504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38848654 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚749853575965696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181548642304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45386962 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181889981440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4899 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088‘lÎW2¥- _wandbruntime‚4211  _runtime‚4211E2ÛÎ ¦åM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚740988028190720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5182902615552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169388568833536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10533221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744818927013888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182902615552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396028 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚746163039894168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169388474234368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396028 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114086 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252314 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38856286 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164149269620736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38856286 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚750001829629440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19121852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5239223225856 _runtime‚4211.824731234 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 _step‚402 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19121584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚746162682304806 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829629440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84242321 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252314 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0´@&ð&§êM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5182561269248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45386037 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744818927013888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182902615552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396028 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚746163039894168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169388474234368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396028 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114086 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252314 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38856286 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164149269620736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38856286 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚750001829629440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19121852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5239223225856 _runtime‚4211.824731234 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 _step‚402 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19121584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚746162682304806 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829629440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84242321 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252314 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚740988028190720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5182902615552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169388568833536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10533221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061689 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39230254 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964570624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19309044 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10635440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19309311 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831437 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171076878756864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚757266322949120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5232244144128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39230254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224600097162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39230252 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248970 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8673871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5289513488384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748167051112448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585483264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752033720426496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMyöG²"onitor.on_train_batch_start/segment.all.allocated‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ìaÀç'¶áO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10635440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19309311 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831437 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171076878756864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚757266322949120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5232244144128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39230254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224600097162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39230252 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248970 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8673871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5289513488384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748167051112448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585483264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752033720426496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061689 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39230254 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964570624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19309044 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088•É‹{|2·- _wandbruntime‚4251  _runtime‚4251Õlãë&¸åM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39237886 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39237884 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256771072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚752180961456128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10637518 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974676 _runtime‚4251.975186475 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752180978495488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45840489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165820707547136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088366 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚757414576612864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5233598117376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 _step‚406 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8675616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171111215970304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚757414218227200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39237886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45850480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171111327346688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5290619799552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚753538832385314 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5233598117376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45850480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10637250 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚757414576612864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19313134 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚4949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45840489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088366 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225611150996 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225270600994 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19312863 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078373 ¹?ð&¹êM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078373 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39237886 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39237884 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256771072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚752180961456128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10637518 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974676 _runtime‚4251.975186475 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752180978495488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45840489 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165820707547136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088366 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚757414576612864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5233598117376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 _step‚406 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8675616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171111215970304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚757414218227200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39237886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45850480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171111327346688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5290619799552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚753538832385314 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5233598117376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45850480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10637250 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚757414576612864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19313134 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚4949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45840489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088366 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225611150996 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225270600994 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19312863 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0–K72º- _wandbruntime‚4251  _runtime‚4251eû¬ »ƒ 0loss/FRt1/assignment_loss‚1.6759065352764326 1loss/SRqqt1/detection_loss‚0.9003044478860014 1loss/SRqqt2/detection_loss‚0.6214851192704013 /loss/FBt2/detection_loss‚0.4427974151005722 0loss/FRt2/assignment_loss‚0.7033932125630157 3loss/FBt1/assignment_loss‚0.0006952960500328231 /loss/FRt2/detection_loss‚1.5613398306769044 2loss/SRqqt2/assignment_loss‚0.5118024407961769 2loss/FBt2/assignment_loss‚0.015402625459980664 /loss/FRt1/detection_loss‚0.5636646206937693 .loss/FBt1/detection_loss‚0.805907689165906 epoch‚12 $ _timestamp‚1.784361759901919e+09 _runtime‚4251.977030536 _step‚407 2loss/SRqqt1/assignment_loss‚0.5284084200821693 %loss/total_loss‚8.331107653021363 trainer/global_step‚4949þÅǼˆ 0loss/FRt1/assignment_loss‚1.6759065352764326 1loss/SRqqt1/detection_loss‚0.9003044478860014 1loss/SRqqt2/detection_loss‚0.6214851192704013 /loss/FBt2/detection_loss‚0.4427974151005722 0loss/FRt2/assignment_loss‚0.7033932125630157 3loss/FBt1/assignment_loss‚0.0006952960500328231 /loss/FRt2/detection_loss‚1.5613398306769044 2loss/SRqqt2/assignment_loss‚0.5118024407961769 2loss/FBt2/assignment_loss‚0.015402625459980664 /loss/FRt1/detection_loss‚0.5636646206937693 .loss/FBt1/detection_loss‚0.805907689165906 epoch‚12 $ _timestamp‚1.784361759901919e+09 _runtime‚4251.977030536 _step‚407 2loss/SRqqt1/assignment_loss‚0.5284084200821693 %loss/total_loss‚8.331107653021363 trainer/global_step‚4949—^Õ”Ìb½‚(:ˆ §æìÒØÛ» gpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.0S´ù14792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ ÷ò¾‚(:ç §æìÒêԼ disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 6418.71875* proc.memory.percent‚16.777777671813965 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 6418.71875 disk.sda3.out‚ 274.39453125 network.recv‚ 1.151793e+06# disk.dm-0.out‚12297.28857421875 disk.dm-1.out‚ 274.39453125 disk.rbd1.out‚94.74853515625( disk.nvme0c0n1.out‚12297.28857421875 disk.sda.out‚ 274.39453125 disk.sda3.in‚ 0.00390625 cpu‚26.19322333799026$ disk./.usageGB‚588.9659042358398 disk.drbd2165.in‚0& disk.nvme0n1.out‚12297.28857421875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.00390625( proc.memory.availableMB‚ 483.42578125# memory_percent‚97.0494031906128) disk./.usagePercent‚39.52599600857629 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6418.71875# proc.memory.rssMB‚ 2748.87109375 network.sent‚ 7.733634e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625erÒÔï¿‚(:ä ¶æìÒ—ž‘½' proc.memory.availableMB‚ 483.1796875 proc.cpu.threads‚17 disk.rbd1.out‚94.74853515625' disk.nvme0c0n1.out‚12299.9443359375 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚12299.9443359375$ disk./.usageGB‚588.9660453796387 disk.rbd1.in‚ 0.01953125 disk.dm-0.in‚ 6418.71875 disk.dm-1.in‚ 0.00390625 cpu‚26.19161296635449 network.recv‚ 1.156151e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 6418.71875 disk.sda.in‚ 0.00390625# proc.memory.rssMB‚ 2748.87109375) disk./.usagePercent‚39.52600548085506 disk.sda.out‚ 274.63671875 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 6418.71875 disk.sda3.out‚ 274.63671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚97.05090522766113* proc.memory.percent‚16.777777671813965 disk.sda1.in‚0 network.sent‚ 7.796647e+06% disk.nvme0n1.out‚12299.9443359375 disk.dm-1.out‚ 274.63671875{¯ OÀ‚(:’ ¶æìÒ¹ˆ…¾ gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚269.905' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.98099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä;<“Á‚(:ˆ ÅæìÒ°„¨» gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à8‚ë‚(:à ÅæìÒ¢‹‚½ proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01953125 disk.dm-1.out‚ 274.89453125 disk.sda3.in‚ 0.00390625* proc.memory.percent‚16.777777671813965 disk.sda1.in‚0 disk.dm-0.in‚ 6418.71875 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625$ memory_percent‚97.04818725585938 network.recv‚ 1.158754e+06 disk.rbd0.out‚0# proc.memory.rssMB‚ 2748.87109375 disk.drbd2165.in‚0 disk.rbd1.out‚94.81103515625 disk.sda.out‚ 274.89453125 disk.sda3.out‚ 274.89453125$ disk./.usageGB‚588.9659271240234 disk.sda2.out‚0" disk.dm-0.out‚12304.2919921875' disk.nvme0c0n1.out‚12304.2919921875# proc.memory.availableMB‚483.625 disk.nvme0n1.in‚ 6418.71875% disk.nvme0n1.out‚12304.2919921875 network.sent‚ 7.800177e+06( disk./.usagePercent‚39.5259975446215 disk.nvme0c0n1.in‚ 6418.71875 cpu‚26.190020834132483vfëÊ2Ã- _wandbruntime‚4291  _runtime‚4291­Õ“ÇÄÁ & lr-AdamW/pg2‚0.0003785337051992595 trainer/global_step‚4999 % _timestamp‚1.7843617990108774e+09 _runtime‚4291.086211472 _step‚408 & lr-AdamW/pg1‚0.0003785337051992595Íx‚äÌÅÆ & lr-AdamW/pg1‚0.0003785337051992595 & lr-AdamW/pg2‚0.0003785337051992595 trainer/global_step‚4999 % _timestamp‚1.7843617990108774e+09 _runtime‚4291.086211472 _step‚408˜êÿ@2Æ- _wandbruntime‚4291  _runtime‚4291›« /â'ÇÜO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5340730323968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764679069932544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46295866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167458698102784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5283280985088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5282939645952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8761118 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46295866 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚4999 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897741 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10739452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39611852 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167458716715008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069932544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10739176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172799537713152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329478 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19500294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4291.087506196 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39611852 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711554048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843617990120146e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5340820998144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755492584706048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5283280985088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279Ü•k>ç'ÈáO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5274876034180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46285889 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39611854 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275216584070 _step‚409 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19500573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚759395771908096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764678711554048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907720 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚760767460740228 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10739452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39611852 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167458716715008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069932544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10739176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172799537713152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329478 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19500294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4291.087506196 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39611852 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711554048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843617990120146e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5340820998144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755492584706048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5283280985088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5340730323968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764679069932544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46295866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167458698102784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5283280985088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5282939645952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8761118 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46295866 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚4999 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897741 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19504022 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924418 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39619486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39619484 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172833875447296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784361799625266e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46304932 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46304932 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5341748585984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952272896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294941 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755639095377920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39619484 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚4999 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914425 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚760914982465822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10741159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5341839252992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764827323596288 6xï&ËéM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755639095377920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39619484 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚4999 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914425 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚760914982465822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10741159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5341839252992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764827323596288 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762866 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284293619200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759543012937728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914425 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10741432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚ 4291.70080076 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284293619200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167492145473536 _step‚410 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19504298 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172833984726528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764827323596288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39619486 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19504022 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924418 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39619486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39619484 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172833875447296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784361799625266e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46304932 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46304932 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5341748585984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952272896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294941š“Êfq2Ì- _wandbruntime‚4291  _runtime‚4291ý‡¹‡ ̓ 1loss/SRqqt2/detection_loss‚0.6742376660801737 /loss/FBt2/detection_loss‚0.5421800430744206 % _timestamp‚1.7843617996272066e+09 2loss/FBt2/assignment_loss‚0.020125195768834255 trainer/global_step‚4999 epoch‚12 /loss/FRt1/detection_loss‚0.5503303000877487 /loss/FBt1/detection_loss‚1.1875290419275482 %loss/total_loss‚8.866446165734244 _runtime‚4291.702267338 0loss/FRt1/assignment_loss‚1.7299572809964445 /loss/FRt2/assignment_loss‚0.654823078190463 /loss/FRt2/detection_loss‚1.4194778562270227 1loss/SRqqt1/detection_loss‚0.9084589328020543 _step‚411 2loss/SRqqt1/assignment_loss‚0.6121778134297431 2loss/SRqqt2/assignment_loss‚0.5561922300431966 2loss/FBt1/assignment_loss‚0.010956727106593503œ† Έ _step‚411 2loss/SRqqt1/assignment_loss‚0.6121778134297431 2loss/SRqqt2/assignment_loss‚0.5561922300431966 2loss/FBt1/assignment_loss‚0.010956727106593503 1loss/SRqqt2/detection_loss‚0.6742376660801737 /loss/FBt2/detection_loss‚0.5421800430744206 % _timestamp‚1.7843617996272066e+09 2loss/FBt2/assignment_loss‚0.020125195768834255 trainer/global_step‚4999 epoch‚12 /loss/FRt1/detection_loss‚0.5503303000877487 /loss/FBt1/detection_loss‚1.1875290419275482 %loss/total_loss‚8.866446165734244 _runtime‚4291.702267338 0loss/FRt1/assignment_loss‚1.7299572809964445 /loss/FRt2/assignment_loss‚0.654823078190463 /loss/FRt2/detection_loss‚1.4194778562270227 1loss/SRqqt1/detection_loss‚0.9084589328020543›—Ã_K÷Ï‚(:ì ÔæìÒÑù†¼ network.sent‚ 7.805731e+06! disk.nvme0c0n1.in‚ 6487.515625 disk.drbd2165.out‚0$ memory_percent‚97.04816341400146* proc.memory.percent‚16.777777671813965 disk.dm-0.in‚ 6487.515625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 275.36328125 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚588.9660606384277 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 disk.rbd1.out‚94.81103515625* disk./.usagePercent‚39.526006504885196 disk.sda2.out‚0 network.recv‚ 1.162018e+06 disk.sda.out‚ 275.36328125 disk.sda1.out‚0( proc.memory.availableMB‚ 483.62890625# proc.memory.rssMB‚ 2748.87109375 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 275.36328125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚12307.40087890625 disk.nvme0n1.in‚ 6487.515625& disk.nvme0n1.out‚12307.40087890625# disk.dm-0.out‚12307.40087890625 disk.rbd1.in‚ 0.01953125 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚26.18833555094555t1žÐ‚(:“ ÔæìÒ턼 gpu.0.gpu‚77.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.051' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.410199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V“Ñ‚(:ˆ ãæìÒº¨×º gpu.0.gpu‚94.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸;ìÓöÒ‚(:ë ãæìÒËëž½ disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 6487.515625 disk.nvme0n1.in‚ 6487.515625( proc.memory.availableMB‚ 483.69140625 disk.sda1.in‚0# proc.memory.rssMB‚ 2748.87109375 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6487.515625( disk.nvme0c0n1.out‚12312.94873046875 cpu‚26.186749446322395 network.sent‚ 7.868812e+06 network.recv‚ 1.166603e+06 disk.drbd2165.out‚0# disk.dm-0.out‚12312.94873046875# memory_percent‚97.0477819442749* proc.memory.percent‚16.777777671813965 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 275.55859375 disk.rbd1.in‚ 0.01953125) disk./.usagePercent‚39.52600010469684 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 275.55859375 disk.rbd0.out‚0& disk.nvme0n1.out‚12312.94873046875 disk.sda.out‚ 275.55859375 disk.sda3.in‚ 0.00390625$ disk./.usageGB‚588.9659652709961 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚94.88916015625}Tg2Ó- _wandbruntime‚4331  _runtime‚4331‡ÄÔ¾ trainer/global_step‚5049 $ _timestamp‚1.784361838948922e+09 _runtime‚4331.024253443 _step‚412 % lr-AdamW/pg1‚0.000377356863995059 % lr-AdamW/pg2‚0.000377356863995059ÓÔÉÕà % lr-AdamW/pg1‚0.000377356863995059 % lr-AdamW/pg2‚0.000377356863995059 trainer/global_step‚5049 $ _timestamp‚1.784361838948922e+09 _runtime‚4331.024253443 _step‚412œáÍÌ 2Ö- _wandbruntime‚4331  _runtime‚4331©ÐÌ«â'×ÜO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768143610820736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5391953363456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174522196581888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169130154641408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5325833070978 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39993452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10842661 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843618389499934e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5333976486912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766757840429056 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5392041940480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚413 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8848368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733793 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174522089392640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10842919 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚762818118299648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5333635147776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816915968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5325492521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46740341 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19691290 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458537472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 5547581442^ØáO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39993454 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19691029 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚766757840429056 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5333976486912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39993454 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46740341 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848371 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743772 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766757823389696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚4331.025489647 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5333635147776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46750318 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768143968409986 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚762818135339008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772091458537472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768143610820736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5391953363456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174522196581888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169130154641408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5325833070978 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39993452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10842661 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843618389499934e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pooîÌM,äl.allocated‚ 5333976486912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766757840429056 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5392041940480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚413 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8848368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733793 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174522089392640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10842919 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚762818118299648 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5333635147776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816915968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5325492521088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46740341 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19691290 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458537472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760470 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10844691 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40001086 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46749393 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19694804 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5393065968640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5334989121024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚772239712194048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚5049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174556535895040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326503574810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚766905064419328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766905081458688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19695081 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5334647774720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚768291490135692 _runtime‚4331.637365333 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46759384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5334989121024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326844124812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚772240070579712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762964646010880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8850116 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40001084 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86750477 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40001086 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762964628971520 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169163564787712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46749393 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ö5ïñ&ÛëM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86750477 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760470 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10844691 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40001086 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46749393 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19694804 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5393065968640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5334989121024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚772239712194048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚5049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174556535895040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326503574810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚766905064419328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766905081458688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19695081 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5334647774720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚768291490135692 _runtime‚4331.637365333 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46759384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5334989121024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326844124812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚772240070579712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762964646010880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8850116 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40001084 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86750477 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40001086 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762964628971520 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169163564787712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46749393 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169163583399936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5392971107328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚768291132546330 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5334647774720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843618395618572e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚766905081458688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174556649368576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PIK±ùä‚(:î çìÒü‚ʼ disk.dm-1.in‚ 0.00390625 network.sent‚ 7.941715e+06 disk.sda1.in‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚12327.5263671875 disk.sda3.out‚ 276.52734375) disk./.usagePercent‚39.52597732002629 disk.sda1.out‚0 disk.dm-1.out‚ 276.52734375 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚95.01416015625 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6556.31640625 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.00390625# disk.nvme0c0n1.in‚ 6556.31640625 disk.sda3.in‚ 0.00390625' proc.memory.availableMB‚ 483.1796875# proc.memory.rssMB‚ 2748.87109375* proc.memory.percent‚16.777777671813965$ disk./.usageGB‚588.9656257629395 proc.cpu.threads‚17 network.recv‚ 1.177528e+06" disk.dm-0.out‚12327.5263671875 cpu‚26.181938220886476 disk.sda2.in‚0 disk.dm-0.in‚ 6556.31640625 disk.rbd0.out‚0 disk.sda.out‚ 276.52734375' disk.nvme0c0n1.out‚12327.5263671875$ memory_percent‚97.05090522766113<ü2å- _wandbruntime‚4370  _runtime‚43703UùÆæÀ % _timestamp‚1.7843618789162424e+09 _runtime‚ 4370.99154955 _step‚416 & lr-AdamW/pg1‚0.0003761697554760086 & lr-AdamW/pg2‚0.0003761697554760086 trainer/global_step‚5099Ñ[¢“ËçÅ % _timestamp‚1.7843618789162424e+09 _runtime‚ 4370.99154955 _step‚416 & lr-AdamW/pg1‚0.0003761697554760086 & lr-AdamW/pg2‚0.0003761697554760086 trainer/global_step‚5099 ¾æ]‚2è- _wandbruntime‚4370  _runtime‚4370=yû¶éÜO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 % _timestamp‚1.7843618789173064e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774119891910656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚774119874871296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204770 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881758 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770143668932608 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_tòÏ9,#rain_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176244701913600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10946424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40375052 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚770143651893248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5384330649600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8935618 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚779504563899392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330649600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚4370.992761931 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5443127958016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775520118490494 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19882045 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40375054 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170801573955584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚779504205520896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205520896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5384671988736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47194793 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8935621 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5384671988736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170801592567808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47194793 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176244811200000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87569845 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775519760901244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚417 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚779504563899392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40375054 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144òT'àç'êáO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47194793 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176244811200000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87569845 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775519760901244 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚417 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚779504563899392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40375054 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 % _timestamp‚1.7843618789173064e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774119891910656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚774119874871296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204770 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881758 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770143668932608 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176244701913600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10946424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40375052 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚770143651893248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5384330649600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8935618 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚779504563899392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330649600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚4370.992761931 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5443127958016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775520118490494 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19882045 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40375054 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170801573955584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚779504205520896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205520896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5384671988736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47194793 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8935621 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5384671988736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170801592567808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¡š¤Ú2ë- _wandbruntime‚4371  _runtime‚4371 ¶x§ì&ìæM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40382686 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774267115900928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8937363 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774267132940288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5385684622848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596522 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10948212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚5099 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8937366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170835002714112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40382686 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40382684 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779652459177472 _step‚418 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775667640216200 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚4371.604485465 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203845 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596522 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19885850 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19885575 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5377120061718 % _timestamp‚1.7843618795289764e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40382684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5377460611720 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779652817563136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684622848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5444147440640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚779652817563136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47203845 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775667282626838 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983${Æ$¬ íëM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779652817563136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684622848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5444147440640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚779652817563136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47203845 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775667282626838 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40382686 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774267115900928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8937363 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774267132940288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5385684622848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596522 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10948212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚5099 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8937366 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170835002714112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40382686 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40382684 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779652459177472 _step‚418 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775667640216200 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚4371.604485465 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203845 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596522 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19885850 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19885575 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5377120061718 % _timestamp‚1.7843618795289764e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40382684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5377460611720 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¢†2î- _wandbruntime‚4371  _runtime‚4371 {^• ï„ _runtime‚4371.606934804 0loss/FBt2/detection_loss‚0.39195887103294397 epoch‚12 trainer/global_step‚5099 % _timestamp‚1.7843618795318868e+09 _step‚419 2loss/FBt2/assignment_loss‚0.022513482176434074 /loss/FRt2/detection_loss‚1.5484045795320966 /loss/FBt1/detection_loss‚0.9967409394405506 0loss/FRt2/assignment_loss‚0.6516858139676911 2loss/SRqqt1/assignment_loss‚0.6781787525958518 /loss/FRt1/detection_loss‚0.5071995156574141 0loss/SRqqt1/detection_loss‚1.041913566104677 %loss/total_loss‚8.927455931432956 0loss/FRt1/assignment_loss‚1.9218871426183495 2loss/SRqqt2/assignment_loss‚0.5126012312982097 3loss/FBt1/assignment_loss‚0.0012179762647237854 0loss/SRqqt2/detection_loss‚0.65315406074401434¨<ð‰ 0loss/FBt2/detection_loss‚0.39195887103294397 epoch‚12 trainer/global_step‚5099 % _timestamp‚1.7843618795318868e+09 _step‚419 2loss/FBt2/assignment_loss‚0.022513482176434074 /loss/FRt2/detection_loss‚1.5484045795320966 /loss/FBt1/detection_loss‚0.9967409394405506 0loss/FRt2/assignment_loss‚0.6516858139676911 2loss/SRqqt1/assignment_loss‚0.6781787525958518 /loss/FRt1/detection_loss‚0.5071995156574141 0loss/SRqqt1/detection_loss‚1.041913566104677 %loss/total_loss‚8.927455931432956 0loss/FRt1/assignment_loss‚1.9218871426183495 2loss/SRqqt2/assignment_loss‚0.5126012312982097 3loss/FBt1/assignment_loss‚0.0012179762647237854 0loss/SRqqt2/detection_loss‚0.653154060744014 _runtime‚4371.606934804£±Zñžñ‚(:“ Ÿçì҇쟺 gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Ajõò‚(:ê ŸçìÒ¢¯¥½ disk.rbd1.out‚95.09228515625 cpu‚26.18034480054982 disk.sda3.out‚ 276.6953125 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚12334.07763671875 disk.dm-1.in‚ 0.00390625 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125# disk.nvme0c0n1.in‚ 6556.31640625# proc.memory.rssMB‚ 2748.87109375! disk.nvme0n1.in‚ 6556.31640625& disk.nvme0n1.out‚12334.07763671875$ proc.memory.availableMB‚482.9375 disk.drbd2165.out‚0 network.recv‚ 1.181105e+06 disk.drbd2165.in‚0 disk.dm-0.in‚ 6556.31640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12334.07763671875* proc.memory.percent‚16.777777671813965$ disk./.usageGB‚588.9651870727539 disk.dm-1.out‚ 276.6953125) disk./.usagePercent‚39.52594787915986$ memory_percent‚97.05238342285156 disk.rbd0.in‚0 network.sent‚ 7.946916e+06 disk.sda.out‚ 276.6953125 disk.sda3.in‚ 0.00390625ꧬ‘ó‚(:† ®çìÒ†àšº gpu.0.gpu‚9.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚216.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.2028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 &úô‚(:ï ®çìÒ퀼 disk.sda3.in‚ 0.00390625 disk.rbd1.out‚95.09228515625& disk.nvme0n1.out‚12337.05029296875$ memory_percent‚97.05064296722412* proc.memory.percent‚16.777777671813965# disk.dm-0.out‚12337.05029296875 network.recv‚ 1.185202e+06 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6556.31640625( disk.nvme0c0n1.out‚12337.05029296875 disk.sda3.out‚ 277.0078125 disk.drbd2165.in‚0 disk.sda.in‚ 0.00390625( proc.memory.availableMB‚ 483.22265625) disk./.usagePercent‚39.52597168786054$ disk./.usageGB‚588.9655418395996 disk.drbd2165.out‚0 disk.dm-0.in‚ 6556.31640625# proc.memory.rssMB‚ 2748.87109375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 277.0078125 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6556.31640625 cpu‚26.178786953036262 disk.sda.out‚ 277.0078125 network.sent‚ 8.009842e+06 disk.sda1.in‚0 disk.rbd0.out‚0PVĪõ‚(:’ ½çìÒ¦§±¼ gpu.0.gpu‚27.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e¶Cõö‚(:ê ½çìÒŸÔÒ¾ disk.sda2.out‚0 network.sent‚ 8.014825e+06* disk./.usagePercent‚39.525973479913276# disk.nvme0n1.out‚12341.57421875 disk.sda3.out‚ 277.37109375 cpu‚26.177247437319206 disk.sda1.out‚0 disk.dm-0.out‚12341.57421875 disk.rbd1.in‚ 0.01953125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 6556.31640625$ memory_percent‚97.05111980438232( proc.memory.availableMB‚ 483.14453125 network.recv‚ 1.188838e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625% disk.nvme0c0n1.out‚12341.57421875* proc.memory.percent‚16.777777671813965$ disk./.usageGB‚588.9655685424805! disk.nvme0n1.in‚ 6556.31640625 disk.dm-0.in‚ 6556.31640625 disk.dm-1.out‚ 277.37109375# proc.memory.rssMB‚ 2748.87109375 disk.sda.out‚ 277.37109375 disk.rbd1.out‚95.15478515625£ža2÷- _wandbruntime‚4410  _runtime‚44101’ Éøà ' lr-AdamW/pg2‚0.00037497245588787026 trainer/global_step‚5149 % _timestamp‚1.7843619186357253e+09 _runtime‚4410.711145112 _step‚420 ' lr-AdamW/pg1‚0.00037497245588787026'Ç eÎùÈ ' lr-AdamW/pg1‚0.00037497245588787026 ' lr-AdamW/pg2‚0.00037497245588787026 trainer/global_step‚5149 % _timestamp‚1.7843619186357253e+09 _runtime‚4410.711145112 _step‚420¤!èÒ2ú- _wandbruntime‚4410  _runtime‚4410®ã'ûÝO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40756654 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786917310882816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044794 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚421 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435367490560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952504320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367490560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786916952504320 trainer/global_step‚5149 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659222 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20072423 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5494298136576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177967310018560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405897 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5435026151424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20072700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9022871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11049555 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843619186369412e+09 _runtime‚4410.712430819 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47649245 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47659222 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979Ar%üâO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5494399296512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40756654 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786917310882816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044794 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚421 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435367490560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952504320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367490560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786916952504320 trainer/global_step‚5149 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659222 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20072423 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5494298136576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177967310018560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405897 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5435026151424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20072700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9022871 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11049555 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843619186369412e+09 _runtime‚4410.712430819 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47649245 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47659222 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20076143 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚4411.327252777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚781629167382528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777615713198080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40764284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11051800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707346 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5436380124672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178001756636160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚781629167382528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20076416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5495315995648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚422 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5436380124672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178001874304000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47658297 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47668288 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40764286 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 trainer/global_step‚5149 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432574 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40764286 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172506459252736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843619192516918e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 trainer/global_step‚5149 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432574 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40764286 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172506459252736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843619192516918e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20076143 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚4411.327252777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚781629167382528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777615713198080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40764284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11051800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707346 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5436380124672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178001756636160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚781629167382528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20076416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5495315995648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚422 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5436380124672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178001874304000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47658297 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438¦¥#ø)2€- _wandbruntime‚4411  _runtime‚4411Ê;Ô¥ ƒ 0loss/FRt1/assignment_loss‚1.7883853317429566 2loss/SRqqt2/assignment_loss‚0.5210282534485187 3loss/FBt1/assignment_loss‚0.0020325503255784433 0loss/SRqqt1/detection_loss‚0.922583453316317 1loss/SRqqt2/detection_loss‚0.5776091615361078 trainer/global_step‚5149 $ _timestamp‚1.784361919253665e+09 2loss/FBt2/assignment_loss‚0.009083549978864033 /loss/FRt2/detection_loss‚1.3872082497099745 epoch‚12 _runtime‚4411.328717522 /loss/FBt1/detection_loss‚0.9844712034266975 /loss/FBt2/detection_loss‚0.5787705909072732 0loss/FRt2/assignment_loss‚0.6702918114453337 2loss/SRqqt1/assignment_loss‚0.6323792724054058 /loss/FRt1/detection_loss‚0.5264362868105799 _step‚423 %loss/total_loss‚8.600279715053606±º¬‚ˆ 0loss/FRt1/assignment_loss‚1.7883853317429566 2loss/SRqqt2/assignment_loss‚0.5210282534485187 3loss/FBt1/assignment_loss‚0.0020325503255784433 0loss/SRqqt1/detection_loss‚0.922583453316317 1loss/SRqqt2/detection_loss‚0.5776091615361078 trainer/global_step‚5149 $ _timestamp‚1.784361919253665e+09 2loss/FBt2/assignment_loss‚0.009083549978864033 /loss/FRt2/detection_loss‚1.3872082497099745 epoch‚12 _runtime‚4411.328717522 /loss/FBt1/detection_loss‚0.9844712034266975 /loss/FBt2/detection_loss‚0.5787705909072732 0loss/FRt2/assignment_loss‚0.6702918114453337 2loss/SRqqt1/assignment_loss‚0.6323792724054058 /loss/FRt1/detection_loss‚0.5264362868105799 _step‚423 %loss/total_loss‚8.600279715053606§™ÝIIžƒ‚(:“ ÌçìÒŠñšº gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.717200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…°Œ÷„‚(:ì ÌçìÒå disk.drbd2165.out‚0# disk.dm-0.out‚12345.66552734375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12345.66552734375 disk.sda2.in‚0 disk.rbd1.in‚ 0.01953125$ memory_percent‚97.05142974853516 disk.nvme0n1.in‚ 6660.9921875 disk.sda.out‚ 277.68359375 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 2748.87109375% proc.memory.availableMB‚ 483.09375 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚26.175614050082174 network.sent‚ 8.018595e+06 network.recv‚ 1.191904e+06* proc.memory.percent‚16.777777671813965) disk./.usagePercent‚39.52597988010164$ disk./.usageGB‚588.9656639099121 disk.dm-1.out‚ 277.68359375 disk.rbd1.out‚95.15478515625" disk.nvme0c0n1.in‚ 6660.9921875 disk.sda3.out‚ 277.68359375 disk.dm-0.in‚ 6660.9921875& disk.nvme0n1.out‚12345.66552734375 disk.sda.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0l¾TÄ“…‚(:ˆ ÛçìҺضº gpu.0.gpu‚93.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%/É÷†‚(:ì ÛçìÒ•çż" disk.nvme0c0n1.in‚ 6660.9921875% proc.memory.availableMB‚ 483.40625 disk.rbd1.in‚ 0.01953125 disk.rbd0.out‚0 disk.sda2.in‚0# disk.dm-0.out‚12352.40673828125$ memory_percent‚97.04952239990234# proc.memory.rssMB‚ 2748.87109375 network.recv‚ 1.196515e+06 disk.rbd1.out‚95.23291015625 disk.sda2.out‚0 disk.sda.out‚ 278.06640625$ disk./.usageGB‚588.9656715393066 disk.nvme0n1.in‚ 6660.9921875 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0* proc.memory.percent‚16.777777671813965 disk.rbd0.in‚0& disk.nvme0n1.out‚12352.40673828125 disk.drbd2165.in‚0 network.sent‚ 8.082757e+06( disk.nvme0c0n1.out‚12352.40673828125 disk.dm-0.in‚ 6660.9921875 proc.cpu.threads‚17) disk./.usagePercent‚39.52598039211671 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 278.06640625 disk.sda1.in‚0 disk.drbd2165.out‚0 cpu‚26.174066214959144 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 278.066406251(gW2‡- _wandbruntime‚4450  _runtime‚4450× ÌYɈà ' lr-AdamW/pg1‚0.00037376504213095917 ' lr-AdamW/pg2‚0.00037376504213095917 trainer/global_step‚5199 % _timestamp‚1.7843619586592276e+09 _runtime‚4450.734518473 _step‚424³ÚñÎ‰È % _timestamp‚1.7843619586592276e+09 _runtime‚4450.734518473 _step‚424 ' lr-AdamW/pg1‚0.00037376504213095917 ' lr-AdamW/pg2‚0.00037376504213095917 trainer/global_step‚5199¨³Ü+)2Š- _wandbruntime‚4450  _runtime‚4450ú=a4¨‹ÜO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚794330057866240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 _step‚425 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179690040466432 _runtime‚4450.739079645 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9110121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784794736119808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48113674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48103697 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41138254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179689931180032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5545572045824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11152840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699487744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11152577 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5545481371648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5485721653248 trainer/global_step‚5199 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚788843994873856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11152840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699487744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11152577 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5545481371648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5485721653248 trainer/global_step‚5199 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚788843994873856 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784794736119808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48113674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0©˜!!2- _wandbruntime‚4451  _runtime‚4451‹ª)øë&ŽåM ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48122740 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚426 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚794478311529984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9111866 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179724496536064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20266752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179724399839744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268626 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5487075626496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784941246791680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚794478311529984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚788991235903488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5546521272832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41145886 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11154886 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20266485 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784941229752320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5478693585536 _runtime‚4451.348600138 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174177878566912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953144320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5486734280192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41145884 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5486734280192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268626 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41145886 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48112749 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5487075626496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41145884 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790419940377216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48122740 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258633 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088¢ýŸ$êM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11154622 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11154886 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20266485 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784941229752320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5478693585536 _runtime‚4451.348600138 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174177878566912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953144320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5486734280192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41145884 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5486734280192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268626 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41145886 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48112749 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5487075626496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41145884 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790419940377216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48122740 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258633 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48122740 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚426 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚794478311529984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9111866 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179724496536064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20266752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179724399839744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268626 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5487075626496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784941246791680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚794478311529984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚788991235903488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 ]KDeviceStatsMonitor.on_train_batch_en¶ÊµŸQd/inactive_split_bytes.small_pool.freed‚ 5546521272832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41145886 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EÒÇ2- _wandbruntime‚4451  _runtime‚4451ÿ¸íÍ‘ 0loss/FRt1/assignment_loss‚1.7550791440556734 0loss/FRt2/assignment_loss‚0.6843232273426408 2loss/FBt2/assignment_loss‚0.014401638742767618 %loss/total_loss‚8.237064997623417 epoch‚12 _step‚427 1loss/FBt1/assignment_loss‚0.02034611885669998 1loss/SRqqt1/detection_loss‚0.9345145546714358 1loss/SRqqt2/detection_loss‚0.6229666042638409 -loss/FBt1/detection_loss‚0.73927671086664 /loss/FBt2/detection_loss‚0.3974821264262365 _runtime‚4451.350096192 2loss/SRqqt2/assignment_loss‚0.5445183411057595 /loss/FRt1/detection_loss‚0.5170430295394596 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚0.4989535848797452 /loss/FRt2/detection_loss‚1.5081599168725175 % _timestamp‚1.7843619592750068e+09å1iw ’† 1loss/FBt1/assignment_loss‚0.02034611885669998 1loss/SRqqt1/detection_loss‚0.9345145546714358 1loss/SRqqt2/detection_loss‚0.6229666042638409 -loss/FBt1/detection_loss‚0.73927671086664 /loss/FBt2/detection_loss‚0.3974821264262365 _runtime‚4451.350096192 2loss/SRqqt2/assignment_loss‚0.5445183411057595 /loss/FRt1/detection_loss‚0.5170430295394596 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚0.4989535848797452 /loss/FRt2/detection_loss‚1.5081599168725175 % _timestamp‚1.7843619592750068e+09 0loss/FRt1/assignment_loss‚1.7550791440556734 0loss/FRt2/assignment_loss‚0.6843232273426408 2loss/FBt2/assignment_loss‚0.014401638742767618 %loss/total_loss‚8.237064997623417 epoch‚12 _step‚427«•Ë‚(:’ êçìÒ󺋻 gpu.0.gpu‚59.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚285.199' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.03980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+*Kô”‚(:é êçìÒŽ­æ¼ disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 278.26953125 proc.cpu.threads‚17 network.sent‚ 8.08778e+06' disk.nvme0c0n1.out‚12357.1611328125 disk.sda.in‚ 0.00390625 disk.drbd2165.in‚0 disk.dm-0.in‚ 6660.9921875 disk.dm-1.out‚ 278.26953125# proc.memory.rssMB‚ 2748.87109375 disk.rbd0.out‚0 disk.sda1.in‚0" disk.dm-0.out‚12357.1611328125$ memory_percent‚97.05362319946289 network.recv‚ 1.199588e+06% disk.nvme0n1.out‚12357.1611328125 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 278.26953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.777777671813965" disk.nvme0c0n1.in‚ 6660.9921875 disk.nvme0n1.in‚ 6660.9921875& proc.memory.availableMB‚ 482.734375 cpu‚26.175433545482885) disk./.usagePercent‚39.52603312966876$ disk./.usageGB‚588.9664573669434 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0 disk.rbd1.out‚95.23291015625 disk.drbd2165.out‚0‹#*É2•- _wandbruntime‚4468  _runtime‚4468bÚŒ¾=–¸{ >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7285112738609314 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7597695589065552 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5450000166893005 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7398272156715393 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7791820764541626 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6759727001190186 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3123861849308014 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166668057441711 +particle/specificity‚0.9103182554244995 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18789540231227875 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821192622184753 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20789436995983124 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2976056635379791 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14370474219322205 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5758546590805054 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5894516110420227 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04202847182750702 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_3‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6507365107536316 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03007311001420021 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_2‚0.7822281122207642 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6507365107536316 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9890873432159424 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14850719273090363 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7135046124458313 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14332222938537598 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1600598692893982 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8658215403556824 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6771847009658813 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930915832519531 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 (particle/accuracy‚0.8496403694152832 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973548054695129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863737225532532 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03195106238126755 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9601914286613464 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.733135461807251 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4494553506374359 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8318127989768982 trainer/global_step‚5199 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1858539879322052 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7591584324836731 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8318127989768982 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6844990253448486 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083432674408 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.590360164642334 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15908724069595337 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041865348816 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.644942581653595 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7776464819908142 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03650215268135071 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 % _timestamp‚1.7843619763518493e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25982001423835754 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1846703737974167 *jet/accuracy_2_of_3‚0.5663177371025085 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6917436122894287 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7510582208633423 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.038279443979263306 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04873160645365715 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936894178390503 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2195870578289032 epoch‚12 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9012047648429871 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03559258207678795 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16791492700576782 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6546623706817627 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,jet/accuracy_4_of_4‚0.007130124606192112 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6783183813095093 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6061951518058777 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.725441038608551 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_3_of_3‚0.0700593963265419 _step‚428 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923709034919739 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7303121089935303 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2124730497598648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4733632504940033 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9958623051643372 *jet/accuracy_2_of_2‚0.1515013873577118 *jet/accuracy_1_of_3‚0.9225616455078125 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3816424012184143 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13283179700374603 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7456538677215576 /particle/accuracy_4_of_4‚0.9420744776725769 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17519143223762512 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4575532376766205 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915753602981567 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6484479904174805 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025229550898075104 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.283211305115511 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5308369398117065 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7541247606277466 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18794848024845123 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4647497832775116 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6498812437057495 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.630456805229187 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13361336290836334 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8417080640792847 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1587999165058136 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962294697761536 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9849206805229187 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6419621109962463 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1729070097208023 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4595733880996704 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5317338109016418 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9704971313476562 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4870731830596924 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09288172423839569 particle/accuracy_1_of_1‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03195106238126755 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6772094964981079 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7390689253807068 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6533637046813965 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17384900152683258 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21645644307136536 *jet/accuracy_1_of_1‚0.3119877278804779 particle/accuracy_2_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9826899766921997 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7143046855926514 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7642356157302856 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.691798985004425 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17137449979782104 *jet/accuracy_2_of_4‚0.7924426794052124 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚4468.427819824 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03540709242224693 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18539480865001678 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6892493963241577 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9939749836921692 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7244923114776611 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7140966057777405 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1675153225660324 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7448223829269409 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852485656738281 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8854060173034668 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03622511029243469 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.201704204082489 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17384900152683258 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +particle/sensitivity‚0.6277585029602051 particle/accuracy_4_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7302893400192261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3587647080421448 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658004641532898 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9132329821586609 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5724520683288574 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38456180691719055 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018612219020724297 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24109259247779846 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6333779692649841‡î³ª3—½{ =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9012047648429871 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03559258207678795 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16791492700576782 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6546623706817627 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,jet/accuracy_4_of_4‚0.007130124606192112 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6783183813095093 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6061951518058777 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.725441038608551 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_3_of_3‚0.0700593963265419 _step‚428 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923709034919739 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7303121089935303 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2124730497598648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4733632504940033 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9958623051643372 *jet/accuracy_2_of_2‚0.1515013873577118 *jet/accuracy_1_of_3‚0.9225616455078125 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3816424012184143 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13283179700374603 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7456538677215576 /particle/accuracy_4_of_4‚0.9420744776725769 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17519143223762512 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4575532376766205 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915753602981567 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6484479904174805 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025229550898075104 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.283211305115511 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5308369398117065 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7541247606277466 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18794848024845123 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4647497832775116 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6498812437057495 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.630456805229187 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13361336290836334 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8417080640792847 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1587999165058136 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962294697761536 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9849206805229187 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6419621109962463 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1729070097208023 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4595733880996704 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5317338109016418 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9704971313476562 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4870731830596924 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09288172423839569 particle/accuracy_1_of_1‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03195106238126755 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6772094964981079 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7390689253807068 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6533637046813965 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17384900152683258 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21645644307136536 *jet/accuracy_1_of_1‚0.3119877278804779 particle/accuracy_2_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9826899766921997 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976301789283752 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967042803764343 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7143046855926514 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7642356157302856 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.691798985004425 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17137449979782104 *jet/accuracy_2_of_4‚0.7924426794052124 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚4468.427819824 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03540709242224693 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18539480865001678 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6892493963241577 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9939749836921692 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7244923114776611 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7140966057777405 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1675153225660324 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7448223829269409 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852485656738281 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8854060173034668 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03622511029243469 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.201704204082489 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17384900152683258 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +particle/sensitivity‚0.6277585029602051 particle/accuracy_4_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7302893400192261 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3587647080421448 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.658004641532898 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9132329821586609 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5724520683288574 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38456180691719055 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018612219020724297 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24109259247779846 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6333779692649841 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7285112738609314 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7597695589065552 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5450000166893005 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7398272156715393 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7791820764541626 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6759727001190186 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3123861849308014 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166668057441711 +particle/specificity‚0.9103182554244995 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18789540231227875 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821192622184753 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20789436995983124 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2976056635379791 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14370474219322205 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5758546590805054 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5894516110420227 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04202847182750702 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_3‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6507365107536316 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03007311001420021 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_2‚0.7822281122207642 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6507365107536316 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9890873432159424 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14850719273090363 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7135046124458313 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14332222938537598 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1600598692893982 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8658215403556824 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6771847009658813 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930915832519531 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 (particle/accuracy‚0.8496403694152832 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973548054695129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863737225532532 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03195106238126755 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9601914286613464 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.733135461807251 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4494553506374359 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_ŸÕÕs purity‚0.8318127989768982 trainer/global_step‚5199 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1858539879322052 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7591584324836731 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8318127989768982 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6844990253448486 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083432674408 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.590360164642334 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15908724069595337 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041865348816 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.644942581653595 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7776464819908142 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03650215268135071 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 % _timestamp‚1.7843619763518493e+09 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25982001423835754 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1846703737974167 *jet/accuracy_2_of_3‚0.5663177371025085 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6917436122894287 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7510582208633423 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967042803764343 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.038279443979263306 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04873160645365715 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936894178390503 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2195870578289032 epoch‚12¬)´O“˜‚(:ˆ ùçìÒª§­º gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02±_ô™‚(:é ùçìÒ²Š¼ disk.dm-0.in‚ 6660.9921875" disk.dm-0.out‚12364.0380859375* proc.memory.percent‚16.778206825256348 disk.drbd2165.out‚0 network.sent‚ 8.150658e+06 disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2748.94140625 network.recv‚ 1.204169e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 278.5234375 disk.rbd1.out‚95.29541015625 cpu‚26.186146979674376 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 6660.9921875 disk.sda.out‚ 278.5234375$ memory_percent‚97.05350399017334 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625( proc.memory.availableMB‚ 482.75390625 disk.sda3.out‚ 278.5234375) disk./.usagePercent‚39.52603927384958$ disk./.usageGB‚588.9665489196777 disk.dm-1.in‚ 0.00390625' disk.nvme0c0n1.out‚12364.0380859375 disk.nvme0n1.in‚ 6660.9921875 disk.sda.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125% disk.nvme0n1.out‚12364.0380859375ºÿUžš‚(:“ ˆèìÒ­×é» gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚240.541' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.108200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»7 ú›‚(:ï ˆèìÒ×Áã½& disk.nvme0n1.out‚12368.36279296875 disk.sda3.out‚ 278.9296875 disk.sda2.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.778206825256348 disk.dm-1.in‚ 0.00390625$ memory_percent‚97.05402851104736 disk.rbd0.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2748.94140625( proc.memory.availableMB‚ 482.66796875 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625# disk./.usageGB‚588.966625213623 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚26.184622005218333 network.recv‚ 1.208129e+06( disk.nvme0c0n1.out‚12368.36279296875 disk.sda.out‚ 278.9296875 disk.sda1.in‚0 disk.dm-0.in‚ 6753.69140625! disk.nvme0n1.in‚ 6753.69140625 network.sent‚ 8.208016e+06* disk./.usagePercent‚39.526044394000266 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚95.29541015625# disk.nvme0c0n1.in‚ 6753.69140625# disk.dm-0.out‚12368.36279296875 disk.dm-1.out‚ 278.9296875†uBàýœ‚(:ò —èìÒÓîļ disk.rbd1.out‚95.43603515625! disk.nvme0n1.in‚ 6753.69140625& disk.nvme0n1.out‚12374.55517578125 disk.sda2.in‚0# proc.memory.rssMB‚ 2748.94140625# disk.nvme0c0n1.in‚ 6753.69140625( disk.nvme0c0n1.out‚12374.55517578125 network.recv‚ 1.211913e+06 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.dm-0.out‚12374.55517578125( proc.memory.availableMB‚ 482.41796875 disk.sda2.out‚0 network.sent‚ 8.213013e+06$ disk./.usageGB‚588.9664688110352 disk.sda.out‚ 279.37890625 disk.dm-0.in‚ 6753.69140625 disk.dm-1.in‚ 0.00390625$ memory_percent‚97.05555438995361 proc.cpu.threads‚17) disk./.usagePercent‚39.52603389769136 disk.rbd0.out‚0 disk.sda3.out‚ 279.37890625 disk.sda1.out‚0 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0* proc.memory.percent‚16.778206825256348 cpu‚26.183085456865786 disk.dm-1.out‚ 279.37890625¸±E“‚(:ˆ —èìÒ’ü‘½ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚229.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.9504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æøùx2ž- _wandbruntime‚4507  _runtime‚4507xòéÎÈŸÂ ' lr-AdamW/pg1‚0.00037254759175520466 ' lr-AdamW/pg2‚0.00037254759175520466 trainer/global_step‚5249 $ _timestamp‚1.784362015724156e+09 _runtime‚4507.799479566 _step‚429ýÅ4ÅÍ Ç ' lr-AdamW/pg1‚0.00037254759175520466 ' lr-AdamW/pg2‚0.00037254759175520466 trainer/global_step‚5249 $ _timestamp‚1.784362015724156e+09 _runtime‚4507.799479566 _step‚429­Å6ű2¡- _wandbruntime‚4507  _runtime‚45072–^á'¢ÛO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41640520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5614944199168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48628471 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20487064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41640520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48638448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5614855622144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚799574951671327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11287176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104705536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20487337 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746327040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181446762646016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545678193949 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746327040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48638448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181446655456768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268989 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175831818446848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546018743839 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798135613005824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5554491699712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48628471 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5554491699712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48628471 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20487064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41640520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48638448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5614855622144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚799574951671327 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11287176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104705536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20487337 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746327040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181446762646016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545678193949 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746327040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48638448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181446655456768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268989 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175831818446848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546018743839 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798135613005824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5554491699712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48628471 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888®éÂN‚2¤- _wandbruntime‚4508  _runtime‚4508¹8Ý¥åM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41648150 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181481109604352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚798282854035456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647514 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9201633 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175865228593152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5555162987520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5615881011200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295666 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚803838358369280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5615965386752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚431 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5555504333824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚803837999983616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20490831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11289198 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚794175426560000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2319ù‹x>0607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181481212592128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41648152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20491093 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504333824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚799722473397033 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚798282854035456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798282836996096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5555162987520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9201636 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11289457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295666 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48647514 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546689247671 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48637523 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚4508.413961553 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48637523 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚52496 @Kð&¦êM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5555162987520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9201636 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11289457 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295666 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48647514 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285673 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546689247671 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48637523 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚4508.413961553 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48637523 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41648150 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181481109604352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚798282854035456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647514 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9201633 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175865228593152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5555162987520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5615881011200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295666 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚803838358369280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5615965386752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797673 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚431 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5555504333824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚803837999983616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20490831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11289198 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚794175426560000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181481212592128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41648152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20491093 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504333824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚799722473397033 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚798282854035456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798282836996096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2¯µ×È2§- _wandbruntime‚4508  _runtime‚4508” ‘¨€ 0loss/FRt1/assignment_loss‚1.7170845980573843 0loss/FRt2/assignment_loss‚0.7393668445166696 2loss/SRqqt2/assignment_loss‚0.4622488565119134 0loss/SRqqt1/detection_loss‚0.842770470722749 0loss/FBt2/detection_loss‚0.43407273578833644 /loss/FRt2/detection_loss‚1.4756569537733282 trainer/global_step‚5249 $ _timestamp‚1.784362016340449e+09 1loss/SRqqt1/assignment_loss‚0.564009567869837 2loss/FBt1/assignment_loss‚0.011318072337712084 0loss/FBt2/assignment_loss‚0.0175131287054335 /loss/FRt1/detection_loss‚0.5172317336885556 1loss/SRqqt2/detection_loss‚0.6281090035574893 epoch‚13 _runtime‚4508.415863635 /loss/FBt1/detection_loss‚0.9266164581291377 %loss/total_loss‚8.335998423658545 _step‚432ïÖ' ©… 1loss/SRqqt2/detection_loss‚0.6281090035574893 epoch‚13 _runtime‚4508.415863635 /loss/FBt1/detection_loss‚0.9266164581291377 %loss/total_loss‚8.335998423658545 _step‚432 0loss/FRt1/assignment_loss‚1.7170845980573843 0loss/FRt2/assignment_loss‚0.7393668445166696 2loss/SRqqt2/assignment_loss‚0.4622488565119134 0loss/SRqqt1/detection_loss‚0.842770470722749 0loss/FBt2/detection_loss‚0.43407273578833644 /loss/FRt2/detection_loss‚1.4756569537733282 trainer/global_step‚5249 $ _timestamp‚1.784362016340449e+09 1loss/SRqqt1/assignment_loss‚0.564009567869837 2loss/FBt1/assignment_loss‚0.011318072337712084 0loss/FBt2/assignment_loss‚0.0175131287054335 /loss/FRt1/detection_loss‚0.5172317336885556°’¢ÕY“ª‚(:ˆ ¦èìÒù“Þ» gpu.0.gpu‚44.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lkázù«‚(:î ¦èìÒ½ÍÚ¼& disk.nvme0n1.out‚12377.81201171875# disk./.usageGB‚588.966381072998 disk.rbd0.in‚0 disk.drbd2165.out‚0 network.sent‚ 8.218046e+06) disk./.usagePercent‚39.52602800951807 disk.sda.in‚ 0.00390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 279.7421875# disk.nvme0c0n1.in‚ 6753.69140625( disk.nvme0c0n1.out‚12377.81201171875! disk.nvme0n1.in‚ 6753.69140625 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01953125 disk.sda.out‚ 279.7421875( proc.memory.availableMB‚ 482.69921875 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0$ memory_percent‚97.05383777618408* proc.memory.percent‚16.778206825256348 cpu‚26.181432130151972 disk.rbd1.out‚95.43603515625 disk.dm-0.in‚ 6753.69140625# proc.memory.rssMB‚ 2748.94140625 network.recv‚ 1.214923e+06 disk.sda2.out‚0# disk.dm-0.out‚12377.81201171875 disk.rbd0.out‚0 disk.sda3.out‚ 279.7421875 disk.sda1.in‚0 disk.drbd2165.in‚0¸ùÉý¬‚(:ò µèìÒêíܼ disk.dm-1.out‚ 279.91015625 disk.rbd1.in‚ 0.01953125( proc.memory.availableMB‚ 482.83203125 disk.sda2.out‚0 disk.dm-0.in‚ 6753.69140625* proc.memory.percent‚16.778206825256348 network.sent‚ 8.281141e+06) disk./.usagePercent‚39.52604132190986# disk.nvme0c0n1.in‚ 6753.69140625 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚26.179955966953784 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 279.91015625# disk.dm-0.out‚12382.56884765625$ memory_percent‚97.05302715301514# proc.memory.rssMB‚ 2748.94140625$ disk./.usageGB‚588.9665794372559! disk.nvme0n1.in‚ 6753.69140625 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.00390625& disk.nvme0n1.out‚12382.56884765625 disk.sda2.in‚0 disk.rbd1.out‚95.49853515625 proc.cpu.threads‚17 network.recv‚ 1.219745e+06( disk.nvme0c0n1.out‚12382.56884765625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 279.91015625 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0ëÜÊž­‚(:“ µèìÒþ¹˜½ gpu.0.gpu‚43.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ëò“®‚(:ˆ ÄèìҤ؄º gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚260.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01“Ñù¯‚(:î ÄèìÒù¥ï» disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625! disk.nvme0n1.in‚ 6831.65234375 disk.dm-1.out‚ 280.27734375) disk./.usagePercent‚39.52606820270095 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 8.286186e+06 disk.sda1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2748.94140625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚12386.4833984375* proc.memory.percent‚16.778206825256348 cpu‚26.178390622410042' proc.memory.availableMB‚ 482.7421875 disk.dm-0.in‚ 6831.65234375 disk.rbd1.out‚95.49853515625' disk.nvme0c0n1.out‚12386.4833984375 disk.sda.in‚ 0.00390625$ memory_percent‚97.05357551574707 disk.sda.out‚ 280.27734375 disk.sda3.out‚ 280.27734375 network.recv‚ 1.222911e+06" disk.dm-0.out‚12386.4833984375 disk.rbd1.in‚ 0.01953125# disk.nvme0c0n1.in‚ 6831.65234375$ disk./.usageGB‚588.9669799804688·Ð2°- _wandbruntime‚4547  _runtime‚4547ÈÚÃäDZÁ & lr-AdamW/pg1‚0.0003713201829551694 & lr-AdamW/pg2‚0.0003713201829551694 trainer/global_step‚5299 % _timestamp‚1.7843620554687564e+09 _runtime‚4547.544077248 _step‚433mû{dÌ²Æ _step‚433 & lr-AdamW/pg1‚0.0003713201829551694 & lr-AdamW/pg2‚0.0003713201829551694 trainer/global_step‚5299 % _timestamp‚1.7843620554687564e+09 _runtime‚4547.544077248±`s.š2³- _wandbruntime‚4547  _runtime‚45474â'´ÜO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49092900 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚811102851688960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42022120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493310464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666233570816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811102851688960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49082923 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49082923 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177503237761024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5604845862400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49092900 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5604845862400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801354466521088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801354449481728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666320050688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚4547.548761231 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚5299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805497664487424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805497647448064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115020 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42022118 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811102493310464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11391384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805497647448064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105041 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187201536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9287138 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187201536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843620554697995e+09 _step‚434 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11391118 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596635230747 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚5299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805497664487424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805497647448064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115020 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42022118 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811102493310464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11391384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚805497647448064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105041 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187201536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9287138 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187201536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843620554697995e+09 _step‚434 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11391118 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596635230747 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42022120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493310464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666233570816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976²L‡¥©2¶- _wandbruntime‚4548  _runtime‚4548­¶Äë&·åM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091975 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚801500960153600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131718 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20682065 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11393182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5606199835648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5667346566656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚805644905517056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667270579712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029750 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚805644888477696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5605858489344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177536666519552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚4548.157762401 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚811251105352704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183204031698432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807098623477541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5597305734579 trainer/global_step‚5299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131718 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42029752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9288883 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚811250746967040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843620560822568e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49101966 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199835648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091975 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9288886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168ìkuð&¸êM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667270579712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029750 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚805644888477696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5605858489344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177536666519552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚4548.157762401 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚811251105352704 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183204031698432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807098623477541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5597305734579 trainer/global_step‚5299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131718 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42029752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9288883 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚811250746967040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843620560822568e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49101966 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199835648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49091975 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9288886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091975 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚801500960153600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131718 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20682065 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11393182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5606199835648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5667346566656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚805644905517056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888³©+(2¹- _wandbruntime‚4548  _runtime‚4548P\À ºý .loss/FBt2/detection_loss‚0.415508788785163 0loss/FRt1/assignment_loss‚1.7425893736474307 2loss/SRqqt2/assignment_loss‚0.4960252536368893 0loss/SRqqt2/detection_loss‚0.620910886225901 .loss/FBt1/detection_loss‚0.584987908890792 3loss/SRqqt1/assignment_loss‚0.47289759886336796 1loss/SRqqt1/detection_loss‚0.9779524221325575 %loss/total_loss‚8.425747361262411 epoch‚13 0loss/FRt2/assignment_loss‚0.8380422638472459 1loss/FBt1/assignment_loss‚0.14209662026999384 1loss/FBt2/assignment_loss‚0.03282408063977011 .loss/FRt1/detection_loss‚0.531307883381177 trainer/global_step‚5299 $ _timestamp‚1.784362056084151e+09 .loss/FRt2/detection_loss‚1.570604280942122 _runtime‚4548.159217258 _step‚436áˆ(»‚ _runtime‚4548.159217258 _step‚436 .loss/FBt2/detection_loss‚0.415508788785163 0loss/FRt1/assignment_loss‚1.7425893736474307 2loss/SRqqt2/assignment_loss‚0.4960252536368893 0loss/SRqqt2/detection_loss‚0.620910886225901 .loss/FBt1/detection_loss‚0.584987908890792 3loss/SRqqt1/assignment_loss‚0.47289759886336796 1loss/SRqqt1/detection_loss‚0.9779524221325575 %loss/total_loss‚8.425747361262411 epoch‚13 0loss/FRt2/assignment_loss‚0.8380422638472459 1loss/FBt1/assignment_loss‚0.14209662026999384 1loss/FBt2/assignment_loss‚0.03282408063977011 .loss/FRt1/detection_loss‚0.531307883381177 trainer/global_step‚5299 $ _timestamp‚1.784362056084151e+09 .loss/FRt2/detection_loss‚1.570604280942122´|Þ.&“¼‚(:ˆ ÓèìÒî­Ýº gpu.0.gpu‚40.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.891' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02›Gô½‚(:é ÓèìÒŽ«½ disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚97.05681800842285' proc.memory.availableMB‚ 482.2109375$ disk./.usageGB‚588.9669303894043 disk.sda.in‚ 0.00390625# proc.memory.rssMB‚ 2748.94140625* proc.memory.percent‚16.778206825256348) disk./.usagePercent‚39.52606487460301 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 disk.sda3.out‚ 280.5546875 disk.sda1.out‚0 network.sent‚ 8.291311e+06 network.recv‚ 1.22656e+06 disk.dm-1.out‚ 280.5546875 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚95.57666015625! disk.nvme0n1.in‚ 6831.65234375% disk.nvme0n1.out‚12392.9560546875 disk.sda3.in‚ 0.00390625 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 6831.65234375 disk.sda.out‚ 280.5546875 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 6831.65234375' disk.nvme0c0n1.out‚12392.9560546875 disk.sda2.in‚0 cpu‚26.17691952338566" disk.dm-0.out‚12392.9560546875ZL¤´“¾‚(:ˆ âèìÒÁ’¨º gpu.0.gpu‚51.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öhkê$¿‚(:ð âèìÒ탃½ disk.rb,ôÖ×d0.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2748.94140625# disk.dm-0.out‚12398.24365234375 network.sent‚ 8.354298e+06 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 6831.65234375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚95.57666015625 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.dm-0.in‚ 6831.65234375% proc.memory.availableMB‚ 482.71875 disk.sda1.in‚0$ memory_percent‚97.05371856689453$ disk./.usageGB‚588.9707717895508 disk.sda3.out‚ 280.82421875 disk.dm-1.in‚ 0.00390625* disk./.usagePercent‚39.526322674189984( disk.nvme0c0n1.out‚12398.24365234375 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 1.230933e+06! disk.nvme0n1.in‚ 6831.65234375 disk.sda.out‚ 280.82421875* proc.memory.percent‚16.778206825256348 cpu‚26.175419662361804 disk.rbd1.in‚ 0.01953125& disk.nvme0n1.out‚12398.24365234375 disk.sda.in‚ 0.00390625 disk.dm-1.out‚ 280.82421875$_Ô62À- _wandbruntime‚4587  _runtime‚4587ÍùíÛÅÁ¿ _step‚437 % lr-AdamW/pg1‚0.000370082894565027 % lr-AdamW/pg2‚0.000370082894565027 trainer/global_step‚5349 % _timestamp‚1.7843620954305203e+09 _runtime‚4587.505830943CWÎÊÂÄ % lr-AdamW/pg1‚0.000370082894565027 % lr-AdamW/pg2‚0.000370082894565027 trainer/global_step‚5349 % _timestamp‚1.7843620954305203e+09 _runtime‚4587.505830943 _step‚437µP(M|2Ã- _wandbruntime‚4587  _runtime‚4587òâ'ÄÜO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818515240293888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42403720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5647251717655 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184892139317248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5717463629824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚808679983075328 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42403718 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9374388 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184892250700800 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818515598672384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11495621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11495342 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843620954316943e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179174675687424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179174694299648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5717556401152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚814326894243093 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5646911167765 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941093 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42403720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚812859715969024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91951072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537375 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240293888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882703360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚812859698929664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20869730 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 _runtime‚4587.507192353 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818515598672384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5655541364224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9374391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49537375 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20870012 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚814327251832343 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42403718 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49547352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983i*võç'ÅáO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11495342 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843620954316943e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179174675687424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179174694299648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5717556401152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚814326894243093 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5646911167765 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941093 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42403720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚812859715969024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91951072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537375 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240293888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882703360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚812859698929664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20869730 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 _runtime‚4587.507192353 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818515598672384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5655541364224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9374391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49537375 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20870012 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚814327251832343 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42403718 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49547352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818515240293888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42403720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5647251717655 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184892139317248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5717463629824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚808679983075328 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42403718 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9374388 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184892250700800 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818515598672384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11495621¶|ÜÆO2Æ- _wandbruntime‚4588  _runtime‚4588cì&ÇæM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42411352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852336128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚439 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49556418 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42411350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5718576251392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚813006939959296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20873747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184926585836032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚814474415968687 _runtime‚4588.119732986 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5656895337472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895337472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5718481390080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553991168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818663493950464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957777 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184926699309568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179208104445952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚814474773558049 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843620960441947e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957777 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49556418 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11497357 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5648262771489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49546427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9376133 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42411352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚813006939959296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11497611 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179208123058176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20873490 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136Lº÷ ÈëM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5648262771489 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49546427 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9376133 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42411352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_‰!¨<è#end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚813006939959296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11497611 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179208123058176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20873490 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42411352 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852336128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚439 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49556418 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42411350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5718576251392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚813006939959296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20873747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184926585836032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚814474415968687 _runtime‚4588.119732986 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5656895337472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895337472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5718481390080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553991168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818663493950464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957777 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184926699309568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179208104445952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚814474773558049 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843620960441947e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957777 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49556418 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11497357 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101·™pKô2É- _wandbruntime‚4588  _runtime‚4588‘¢´*Êþ $loss/total_loss‚8.71314044780306 $ _timestamp‚1.784362096046462e+09 2loss/FBt1/assignment_loss‚0.013745737228943624 1loss/FBt2/assignment_loss‚0.00832662963105986 /loss/FRt1/detection_loss‚0.5502287209132777 /loss/FBt2/detection_loss‚0.5000512274382636 trainer/global_step‚5349 _step‚440 /loss/FRt2/detection_loss‚1.4587448254142972 1loss/SRqqt1/detection_loss‚0.8807776125843367 _runtime‚4588.121873978 1loss/SRqqt1/assignment_loss‚0.630417859265148 epoch‚13 /loss/FRt1/assignment_loss‚1.747404562417492 0loss/FRt2/assignment_loss‚0.6823479413986206 2loss/SRqqt2/assignment_loss‚0.4961728108914212 0loss/SRqqt2/detection_loss‚0.707044462976704 /loss/FBt1/detection_loss‚1.0378780576434952ßÆa ˃ 0loss/FRt2/assignment_loss‚0.6823479413986206 2loss/SRqqt2/assignment_loss‚0.4961728108914212 0loss/SRqqt2/detection_loss‚0.707044462976704 /loss/FBt1/detection_loss‚1.0378780576434952 $loss/total_loss‚8.71314044780306 $ _timestamp‚1.784362096046462e+09 2loss/FBt1/assignment_loss‚0.013745737228943624 1loss/FBt2/assignment_loss‚0.00832662963105986 /loss/FRt1/detection_loss‚0.5502287209132777 /loss/FBt2/detection_loss‚0.5000512274382636 trainer/global_step‚5349 _step‚440 /loss/FRt2/detection_loss‚1.4587448254142972 1loss/SRqqt1/detection_loss‚0.8807776125843367 _runtime‚4588.121873978 1loss/SRqqt1/assignment_loss‚0.630417859265148 epoch‚13 /loss/FRt1/assignment_loss‚1.747404562417492¸»sî‚Ì‚(:’ ñèìÒ™ü½º gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.27839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O{3ÐïÍ‚(:ä ñèìÒÒðñ¼ disk.drbd2165.in‚0 disk.dm-1.out‚281.3125 disk.rbd1.in‚ 0.01953125$ memory_percent‚97.05514907836914 disk.sda2.out‚0 disk.sda.in‚ 0.00390625# disk.dm-0.out‚12403.64404296875 disk.rbd1.out‚95.63916015625 cpu‚26.173930071738123& proc.memory.availableMB‚ 482.484375$ disk./.usageGB‚588.9715614318848 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12403.64404296875& disk.nvme0n1.out‚12403.64404296875 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625 proc.cpu.threads‚17 disk.rbd0.out‚0! disk.nvme0n1.in‚ 6831.65234375 disk.sda1.in‚0 network.sent‚ 8.359325e+06# disk.nvme0c0n1.in‚ 6831.65234375 disk.dm-0.in‚ 6831.65234375 disk.sda3.out‚281.3125 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚16.778206825256348 network.recv‚ 1.234635e+06 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 2748.94140625) disk./.usagePercent‚39.52637566774957 disk.sda.out‚281.3125ärc(΂(:’ €éìÒŠ–Ÿ¼ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßjàkìÏ‚(:á €éìÒ̧‹½ disk.sda.out‚ 281.55859375$ memory_percent‚97.05686569213867 proc.cpu.threads‚17$ disk./.usageGB‚588.9716262817383 disk.sda2.in‚0 disk.sda3.out‚ 281.55859375& proc.memory.availableMB‚ 482.203125* proc.memory.percent‚16.778206825256348 network.sent‚ 8.422258e+06 disk.sda2.out‚0 disk.dm-1.out‚ 281.55859375 disk.rbd1.in‚ 0.01953125 disk.nvme0c0n1.in‚6912.625 disk.dm-1.in‚ 0.00390625 cpu‚26.172500110119234 disk.dm-0.in‚6912.625 disk.rbd1.out‚95.63916015625( disk.nvme0c0n1.out‚12406.44873046875& disk.nvme0n1.out‚12406.44873046875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2748.94140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚6912.625) disk./.usagePercent‚39.52638001987766 disk.sda3.in‚ 0.00390625 network.recv‚ 1.238946e+06 disk.drbd2165.in‚0# disk.dm-0.out‚12406.44873046875 disk.sda.in‚ 0.003906250ÁÜíЂ(:â éìҙȼ network.sent‚ 8.427237e+06 disk.drbd2165.out‚0# disk.dm-0.out‚12410.73193359375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12410.73193359375& disk.nvme0n1.out‚12410.73193359375 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 281.99609375& proc.memory.availableMB‚ 482.390625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 281.99609375 disk.dm-0.in‚6912.625 disk.rbd1.out‚95.71728515625* disk./.usagePercent‚39.526393076261904 disk.drbd2165.in‚0$ disk./.usageGB‚588.9718208312988 disk.nvme0n1.in‚6912.625 disk.sda3.out‚ 281.99609375 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125# proc.memory.rssMB‚ 2748.94140625* proc.memory.percent‚16.778206825256348 disk.nvme0c0n1.in‚6912.625$ memory_percent‚97.05572128295898 cpu‚26.170926284208484 proc.cpu.threads‚17 network.recv‚ 1.242187e+06 disk.sda2.in‚0›0æ‘Ñ‚(:† éìÒîÿ¼ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦(Mk2Ò- _wandbruntime‚4627  _runtime‚4627ÃtèÕÈÓ ' lr-AdamW/pg1‚0.00036883580605349877 ' lr-AdamW/pg2‚0.00036883580605349877 trainer/global_step‚5399 $ _timestamp‚1.784362135415499e+09 _runtime‚4627.490809213 _step‚441|0’ÍÔÇ _step‚441 ' lr-AdamW/pg1‚0.00036883580605349877 ' lr-AdamW/pg2‚0.00036883580605349877 trainer/global_step‚5399 $ _timestamp‚1.784362135415499e+09 _runtime‚4627.490809213¹]é'q2Õ- _wandbruntime‚4627  _runtime‚4627Y"‡á'ÖÛO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚820221767450624 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821703401912851 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987277312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5706578205184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820221750411264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚4627.495086187 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816005516668928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186614795378688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323601 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50001804 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚442 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236866048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49991827 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21060697 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825928345655808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825928345655808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5768681764864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180846113613824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5697527654673 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820221767450624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180846132226048 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42785318 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825928345655808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825928345655808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5768681764864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180846113613824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5697527654673 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820221767450624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180846132226048 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42785318 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987277312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5706578205184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820221750411264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚4627.495086187 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816005516668928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186614795378688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323601 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50001804 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚442 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236866048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49991827 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21060697 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5707590839296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42792952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186649245577728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5769703205376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5698538708395 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826076240933888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21064522 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793829 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11601395 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _runtime‚4628.103616129 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5707249492992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049195 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000879 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9463383 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚820368991440896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180879560984576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820369008480256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11601139 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚820369008480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5698879258397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180879542372352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843621360280774e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92803822 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599319552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50000879 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5707590839296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42792950 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42792950 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803822 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42792950 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42792950 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803822 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5707590839296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42792952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186649245577728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5769703205376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5698538708395 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826076240933888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21064522 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793829 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11601395 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _runtime‚4628.103616129 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5707249492992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049195 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000879 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9463383 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚820368991440896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180879560984576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820369008480256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11601139 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚820369008480256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5698879258397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180879542372352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843621360280774e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92803822 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599319552»Ò©2Û- _wandbruntime‚4628  _runtime‚4628ð p Ü€ /loss/FRt2/detection_loss‚1.4239644291126226 1loss/SRqqt2/detection_loss‚0.5742419214098091 _step‚444 2loss/FBt2/assignment_loss‚0.012667635007297479 0loss/SRqqt1/detection_loss‚0.826912262616679 /loss/FBt2/detection_loss‚0.4767818517929472 $ _timestamp‚1.784362136030817e+09 _runtime‚4628.105884782 0loss/FRt1/assignment_loss‚1.8386549863950674 2loss/SRqqt1/assignment_loss‚0.5325153508564351 1loss/FBt1/assignment_loss‚0.12002240112354005 /loss/FRt1/detection_loss‚0.5110786144956837 .loss/FBt1/detection_loss‚1.234031007480647 0loss/FRt2/assignment_loss‚0.7696813415332013 2loss/SRqqt2/assignment_loss‚0.5674399954156303 %loss/total_loss‚8.887991797239561 epoch‚13 trainer/global_step‚5399T¶Ý Ý… 0loss/FRt1/assignment_loss‚1.8386549863950674 2loss/SRqqt1/assignment_loss‚0.5325153508564351 1loss/FBt1/assignment_loss‚0.12002240112354005 /loss/FRt1/detection_loss‚0.5110786144956837 .loss/FBt1/detection_loss‚1.234031007480647 0loss/FRt2/assignment_loss‚0.7696813415332013 2loss/SRqqt2/assignment_loss‚0.5674399954156303 %loss/total_loss‚8.887991797239561 epoch‚13 trainer/global_step‚5399 /loss/FRt2/detection_loss‚1.4239644291126226 1loss/SRqqt2/detection_loss‚0.5742419214098091 _step‚444 2loss/FBt2/assignment_loss‚0.012667635007297479 0loss/SRqqt1/detection_loss‚0.826912262616679 /loss/FBt2/detection_loss‚0.4767818517929472 $ _timestamp‚1.784362136030817e+09 _runtime‚4628.105884782¼KYçÞ‚(:’ žéìÒÿ餺 gpu.0.gpu‚81.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚262.071' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÏ@Ãìß‚(:á žéìÒ•Ñͼ network.recv‚ 1.245597e+06 disk.nvme0c0n1.in‚6912.625( disk.nvme0c0n1.out‚12414.39794921875# memory_percent‚97.0571756362915 cpu‚26.169516995281917# disk./.usageGB‚588.971923828125 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 282.30859375 disk.dm-1.out‚ 282.30859375 network.sent‚ 8.432253e+06 disk.rbd0.in‚0 disk.sda.out‚ 282.30859375 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚95.71728515625 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0n1.in‚6912.625# proc.memory.rssMB‚ 2748.94140625* proc.memory.percent‚16.778206825256348) disk./.usagePercent‚39.52639998846533 disk.sda.in‚ 0.00390625 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚6912.625 proc.cpu.threads‚17& disk.nvme0n1.out‚12414.39794921875 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0# disk.dm-0.out‚12414.39794921875( proc.memory.availableMB‚ 482.15234375‚‰aœà‚(:‘ ­éìÒωɻ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 gÒîá‚(:ã ­éìҨݺ¼ disk.dm-0.in‚ 6912.640625 disk.rbd1.out‚95.77978515625 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 network.recv‚ 1.250029e+06) disk./.usagePercent‚39.52640869272149 disk.rbd0.out‚0 disk.sda.out‚ 282.62109375$ memory_percent‚97.05591201782227# proc.memory.rssMB‚ 2748.94140625 proc.cpu.threads‚17 network.sent‚ 8.495145e+06& disk.nvme0c0n1.out‚12420.478515625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.778206825256348# disk./.usageGB‚588.972053527832! disk.nvme0c0n1.in‚ 6912.640625 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625& proc.memory.availableMB‚ 482.359375 disk.sda3.out‚ 282.62109375 disk.sda2.out‚0! disk.dm-0.out‚12420.478515625 disk.nvme0n1.in‚ 6912.640625 disk.dm-1.out‚ 282.62109375 cpu‚26.168065491186915 disk.rbd1.in‚ 0.01953125$ disk.nvme0n1.out‚12420.478515625”‡;B‘â‚(:† ¼éìÒ„º» gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]J€øã‚(:í ¼éìÒ´Ò·¼$ disk./.usageGB‚588.9720268249512 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.00390625 cpu‚26.166621849441558 network.sent‚ 8.500169e+06 disk.sda2.in‚0% disk.nvme0n1.out‚12425.1923828125 disk.sda3.in‚ 0.00390625( proc.memory.availableMB‚ 482.37109375$ memory_percent‚97.05584049224854 disk.sda1.in‚0" disk.dm-0.out‚12425.1923828125 disk.dm-1.out‚ 283.0234375 disk.rbd0.out‚0 disk.sda.out‚ 283.0234375 disk.dm-0.in‚ 7007.63671875 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7007.63671875 proc.cpu.threads‚17 disk.sda3.out‚ 283.0234375 network.recv‚ 1.253103e+06 disk.rbd1.in‚ 0.01953125# proc.memory.rssMB‚ 2748.94140625* disk./.usagePercent‚39.526406900668746 disk.sda1.out‚0 disk.rbd1.out‚95.77978515625' disk.nvme0c0n1.out‚12425.1923828125* proc.memory.percent‚16.778206825256348 disk.sda2.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7007.63671875 disk.sda.in‚ 0.003906257e[ú2ä- _wandbruntime‚4667  _runtime‚4667„E»¿ÆåÀ $ _timestamp‚1.784362175142332e+09 _runtime‚4667.217626725 _step‚445 & lr-AdamW/pg1‚0.0003675789975187493 & lr-AdamW/pg2‚0.0003675789975187493 trainer/global_step‚5449•/ú[ËæÅ _runtime‚4667.217626725 _step‚445 & lr-AdamW/pg1‚0.0003675789975187493 & lr-AdamW/pg2‚0.0003675789975187493 trainer/global_step‚5449 $ _timestamp‚1.784362175142332e+09½C°2ç- _wandbruntime‚4667  _runtime‚4667ŸQFÁµ èÜO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932367872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833340734260736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43166918 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827583818932224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚2&û-66 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚829079551993359 % _timestamp‚1.7843621751433644e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734260736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚827583801892864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188337391995904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚4667.218821891 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827583818932224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5756932367872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11702547 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273707008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188337499185152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50446279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚5449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11702813 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21251704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613197 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43166918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182517551540224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93623176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5819929032704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5757273707008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚829079194404109 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5748484691471 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50446279 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21251435 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141581 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50456256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932367872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833340734260736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43166918 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827583818932224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚829079551993359 % _timestamp‚1.7843621751433644e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734260736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚827583801892864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188337391995904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚4667.218821891 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827583818932224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5756932367872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11702547 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273707008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188337499185152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50446279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚5449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11702813 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21251704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613197 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43166918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182517551540224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93623176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5819929032704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5757273707008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚829079194404109 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5748484691471 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50446279 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21251435 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141581 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50456256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43174550 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639874 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43174552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9550636 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50455331 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21255274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚447 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629881 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182550980298752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827731059961856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚827731059961856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50465322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚4667.832032371 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚829227073719065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827731042922496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188371949626880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚823477560934400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚833488987917312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21255534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188371844541952 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43174552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827731042922496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843621757564464e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833489346302976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5757944994816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823477577973760 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5749155195303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11704898 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11704641 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93629881 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50455331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5820864243200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5749495745305|Jz «ìëM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11704641 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93629881 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50455331 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5820864243200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5749495745305 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43174550 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639874 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43174552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9550636 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50455331 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21255274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚447 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629881 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5449 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182550980298752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827731059961856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚827731059961856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50465322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚4667.832032371 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚829227073719065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827731042922496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188371949626880 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚823477560934400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚833488987917312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21255534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188371844541952 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43174552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827731042922496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843621757564464e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833489346302976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5757944994816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823477577973760 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5749155195303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11704898¿:'äÔ2í- _wandbruntime‚4667  _runtime‚4667×ð«ïî‚ epoch‚13 2loss/SRqqt1/assignment_loss‚0.5106723424502266 /loss/FRt2/detection_loss‚1.5251878968889028 1loss/SRqqt1/detection_loss‚0.9762703964155853 0loss/FBt2/detection_loss‚0.36842091870494187 _runtime‚ 4667.83357839 0loss/FRt1/assignment_loss‚1.6911246745914248 0loss/FRt2/assignment_loss‚0.7570787354999658 2loss/FBt1/assignment_loss‚0.011725944144799603 1loss/SRqqt2/detection_loss‚0.7278453138330612 $loss/total_loss‚8.09776499951928 trainer/global_step‚5449 % _timestamp‚1.7843621757585094e+09 _step‚448 2loss/SRqqt2/assignment_loss‚0.4896686219666038 2loss/FBt2/assignment_loss‚0.006901211477891927 /loss/FRt1/detection_loss‚0.5505788717636405 .loss/FBt1/detection_loss‚0.482290071782236eú} ï‡ 2loss/FBt1/assignment_loss‚0.011725944144799603 1loss/SRqqt2/detection_loss‚0.7278453138330612 $loss/total_loss‚8.09776499951928 trainer/global_step‚5449 % _timestamp‚1.7843621757585094e+09 _step‚448 2loss/SRqqt2/assignment_loss‚0.4896686219666038 2loss/FBt2/assignment_loss‚0.006901211477891927 /loss/FRt1/detection_loss‚0.5505788717636405 .loss/FBt1/detection_loss‚0.482290071782236 epoch‚13 2loss/SRqqt1/assignment_loss‚0.5106723424502266 /loss/FRt2/detection_loss‚1.5251878968889028 1loss/SRqqt1/detection_loss‚0.9762703964155853 0loss/FBt2/detection_loss‚0.36842091870494187 _runtime‚ 4667.83357839 0loss/FRt1/assignment_loss‚1.6911246745914248 0loss/FRt2/assignment_loss‚0.7570787354999658À±R¼“ð‚(:ˆ ËéìÒùØм gpu.0.gpu‚93.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚259.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00 „øñ‚(:í ËéìÒŒ½ disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 283.33984375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 283.33984375$ proc.memory.availableMB‚482.1875# proc.memory.rssMB‚ 2748.94140625 disk.dm-0.in‚ 7007.63671875 disk.rbd1.in‚ 0.01953125 cpu‚26.16693299734387 network.sent‚ 8.504275e+06 disk.sda2.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 1.255667e+06$ disk./.usageGB‚588.9721221923828 disk.rbd1.out‚95.92041015625$ memory_percent‚97.05696105957031 disk.sda.out‚ 283.33984375# disk.dm-0.out‚12430.72216796875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12430.72216796875* proc.memory.percent‚16.778206825256348 disk.sda.in‚ 0.00390625 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚12430.72216796875 disk.dm-1.in‚ 0.00390625! disk.nvme0n1.in‚ 7007.63671875) disk./.usagePercent‚39.52641330085711 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 7007.63671875+¢”.‘ò‚(:† ÚéìÒ½Òк gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹%lÏ÷ó‚(:ì ÚéìÒùП¼ disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 283.70703125* proc.memory.percent‚16.778206825256348 network.sent‚ 8.567175e+06$ disk./.usageGB‚588.9720306396484 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚ 0.01953125 disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2748.94140625 cpu‚26.167736775860497 disk.rbd0.out‚0 network.recv‚ 1.25974e+06! disk.dm-0.out‚12433.693359375$ disk.nvme0n1.out‚12433.693359375 proc.cpu.threads‚17 disk.dm-1.out‚ 283.70703125 disk.rbd1.out‚95.92041015625 disk.sda3.in‚ 0.00390625* disk./.usagePercent‚39.526407156676285 disk.dm-0.in‚ 7007.63671875& disk.nvme0c0n1.out‚12433.693359375! disk.nvme0n1.in‚ 7007.63671875 disk.sda.out‚ 283.70703125( proc.memory.availableMB‚ 482.18359375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7007.63671875$ memory_percent‚97.05698490142822ýiÔ2ô- _wandbruntime‚4707  _runtime‚4707¸ ³ÇõÁ _step‚449 & lr-AdamW/pg1‚0.0003663125496832424 & lr-AdamW/pg2‚0.0003663125496832424 trainer/global_step‚5499 % _timestamp‚1.7843622156757898e+09 _runtime‚4707.751113038y~µÓÌöÆ & lr-AdamW/pg2‚0.0003663125496832424 trainer/global_step‚5499 % _timestamp‚1.7843622156757898e+09 _runtime‚4707.751113038 _step‚449 & lr-AdamW/pg1‚0.0003663125496832424Á‰bðž2÷- _wandbruntime‚4707  _runtime‚4707¯6€ýá'øÛO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910708 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840753481244160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481244160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21442546 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚830656600895488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628489 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚4707.756183971 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚830656583856128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5807627869696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190059963888640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 % _timestamp‚1.7843622156805606e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43548520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807969208832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459228 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50910708 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21442806 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50900731 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50900731 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5799101178379 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11806665 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834945870413824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5807627869696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5871050416128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807969208832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834945870413824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11806408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5870974422016 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5799101178379 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11806665 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834945870413824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5807627869696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5871050416128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807969208832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834945870413824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11806408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5870974422016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910708 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840753481244160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481244160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21442546 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚830656600895488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628489 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚450 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚4707.756183971 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚830656583856128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5807627869696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190059963888640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 % _timestamp‚1.7843622156805606e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43548520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807969208832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459228 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50910708 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21442806 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169»ûœ 2ú- _wandbruntime‚4708  _runtime‚4708þC¤‡ ûæM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚830803094528000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4708.368245056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5871987097088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210211 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835093111443456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840901734900736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5799771682211 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21446358 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5808981842944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11808475 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835093094404096 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9637886 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21446620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚5499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50919774 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5872075666944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843622162927196e+09 _step‚451 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465933 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5808640496640 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835093094404096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43556152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9637883 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43556152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840902093286400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5808981842944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43556150 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835093094404096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43556152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9637883 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43556152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840902093286400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5808981842944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43556150 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4708.368245056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5871987097088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210211 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835093111443456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840901734900736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5799771682211 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21446358 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5808981842944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11808475 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835093094404096 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9637886 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21446620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚5499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50919774 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5872075666944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843622162927196e+09 _step‚451Ã@:ö2ý- _wandbruntime‚4708  _runtime‚4708M©º þ† /loss/FRt2/detection_loss‚1.5622449594114411 0loss/FBt2/detection_loss‚0.45701860175704234 epoch‚13 % _timestamp‚1.7843622162951744e+09 _runtime‚4708.370241374 0loss/FRt1/assignment_loss‚1.6878637567274553 0loss/FRt2/assignment_loss‚0.6880001654872646 3loss/SRqqt2/assignment_loss‚0.46537360966663005 1loss/SRqqt1/detection_loss‚0.9699196581463588 trainer/global_step‚5499 2loss/SRqqt1/assignment_loss‚0.5256192670691588 1loss/FBt1/assignment_loss‚0.01511757992334708 3loss/FBt2/assignment_loss‚0.0020423975188146185 1loss/SRqqt2/detection_loss‚0.7056753527373075 /loss/FBt1/detection_loss‚0.8943647797529896 _step‚452 /loss/FRt1/detection_loss‚0.5532221007758271 %loss/total_loss‚8.526462228973637äÁ¥†ÿ‹ /loss/FRt2/detection_loss‚1.5622449594114411 0loss/FBt2/detection_loss‚0.45701860175704234 epoch‚13 % _timestamp‚1.7843622162951744e+09 _runtime‚4708.370241374 0loss/FRt1/assignment_loss‚1.6878637567274553 0loss/FRt2/assignment_loss‚0.6880001654872646 3loss/SRqqt2/assignment_loss‚0.46537360966663005 1loss/SRqqt1/detection_loss‚0.9699196581463588 trainer/global_step‚5499 2loss/SRqqt1/assignment_loss‚0.5256192670691588 1loss/FBt1/assignment_loss‚0.01511757992334708 3loss/FBt2/assignment_loss‚0.0020423975188146185 1loss/SRqqt2/detection_loss‚0.7056753527373075 /loss/FBt1/detection_loss‚0.8943647797529896 _step‚452 /loss/FRt1/detection_loss‚0.5532221007758271 %loss/total_loss‚8.526462228973637ÄœtUBó€‚(:è ééìÒˆ¥½ disk.drbd2165.in‚0$ proc.memory.availableMB‚481.6875 network.sent‚ 8.572298e+06# disk.nvme0c0n1.in‚ 7007.63671875 cpu‚26.16825269870378 network.recv‚ 1.263517e+06 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.dm-0.in‚ 7007.63671875 disk.dm-1.out‚ 284.109375 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 284.109375# disk.dm-0.out‚12438.02099609375 disk.dm-1.in‚ 0.00390625# proc.memory.rssMB‚ 2748.94140625$ disk./.usageGB‚588.9720153808594 disk.drbd2165.out‚0 disk.rbd1.out‚95.98291015625! disk.nvme0n1.in‚ 7007.63671875 disk.sda.out‚ 284.109375 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚97.06001281738281* disk./.usagePercent‚39.526406132646144 proc.cpu.threads‚17( disk.nvme0c0n1.out‚12438.02099609375* proc.memory.percent‚16.778206825256348 disk.rbd1.in‚ 0.01953125 disk.rbd0.out‚0& disk.nvme0n1.out‚12438.02099609375 disk.sda1.out‚0R‚’‚(:‡ ééìÒÕ™ò½ gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚252.925' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SL%|“‚‚(:ˆ øéìÒ¡Ìκ gpu.0.gpu‚55.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾|_ öƒ‚(:ë øéìÒß¼ê¼* proc.memory.percent‚16.778206825256348* disk./.usagePercent‚39.526407156676285$ disk./.usageGB‚588.9720306396484 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 284.36328125 network.sent‚ 8.635242e+06" disk.nvme0c0n1.in‚ 7020.4609375' disk.nvme0c0n1.out‚12441.2900390625 disk.nvme0n1.in‚ 7020.4609375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 284.36328125 network.recv‚ 1.267969e+06" disk.dm-0.out‚12441.2900390625" memory_percent‚97.060227394104( proc.memory.availableMB‚ 481.65234375 proc.cpu.threads‚17% disk.nvme0n1.out‚12441.2900390625 disk.sda3.in‚ 0.00390625 cpu‚26.167933850943555 disk.sda2.out‚0 disk.dm-0.in‚ 7020.4609375 disk.rbd1.out‚95.98291015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01953125# proc.memory.rssMB‚ 2748.94140625 disk.drbd2165.out‚0 disk.dm-1.out‚ 284.36328125 disk.rbd0.out‚0Š¨iõž„‚(:“ ‡êìÒô¸Ðº gpu.0.gpu‚78.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚208.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.693799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øaw9ú…‚(:ï ‡êìÒ£¾‡½ network.sent‚ 8.640322e+06 network.recv‚ 1.271389e+06 disk.drbd2165.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.778206825256348 cpu‚26.167087825175805 disk.sda1.out‚0 disk.rbd1.out‚96.06103515625% proc.memory.availableMB‚ 482.09375 disk.dm-1.in‚ 0.00390625$ disk./.usageGB‚588.9720687866211 disk.dm-0.in‚ 7093.12890625# disk.nvme0c0n1.in‚ 7093.12890625( disk.nvme0c0n1.out‚12444.84130859375 disk.sda.in‚ 0.00390625) disk./.usagePercent‚39.52640971675162 disk.sda1.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 7093.12890625 proc.cpu.threads‚17# disk.dm-0.out‚12444.84130859375& disk.nvme0n1.out‚12444.84130859375 disk.sda.out‚ 284.79296875 disk.dm-1.out‚ 284.79296875 disk.sda2.in‚0 disk.rbd1.in‚ 0.01953125 disk.sda3.in‚ 0.00390625$ memory_percent‚97.05753326416016 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 284.79296875# proc.memory.rssMB‚ 2748.94140625p~ÕF2†- _wandbruntime‚4748  _runtime‚4748.b LJÁ ' lr-AdamW/pg1‚0.00036503654388855584 ' lr-AdamW/pg2‚0.00036503654388855584 trainer/global_step‚5549 $ _timestamp‚1.784362256602812e+09 _runtime‚ 4748.67823702 _step‚453%nÇÇÌˆÆ _runtime‚ 4748.67823702 _step‚453 ' lr-AdamW/pg1‚0.00036503654388855584 ' lr-AdamW/pg2‚0.00036503654388855584 trainer/global_step‚5549 $ _timestamp‚1.784362256602812e+09Å•ÒXý2‰- _wandbruntime‚4748  _runtime‚47482ãGìÇ"ŠÛO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚848166586606080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21633700 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837982134489088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11910312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51355183 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842307921895424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5858323371520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚837982117449728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665287 trainer/global_step‚5549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11910592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784362256603814e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9723391 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842307904856064 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚848166228227584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚842307904856064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185860427393024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5849377115397 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51365160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191782531947520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185860446005248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5858664710656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚4748.679299345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323371520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21633983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842307921895424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191782647525376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚454 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5858664710656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43930120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228227584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43930118 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43930118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888¡‘)ýæ'‹àO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21633700 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837982134489088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11910312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51355183 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842307921895424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5858323371520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚837982117449728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665287 trainer/global_step‚5549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11910592 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784362256603814e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9723391 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842307904856064 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚848166228227584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚842307904856064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185860427393024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5849377115397 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51365160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191782531947520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185860446005248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5858664710656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚4748.679299345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323371520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21633983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842307921895424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191782647525376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚454 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5858664710656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43930120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228227584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43930118 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43930118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚848166586606080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÆÙ@“02Œ- _wandbruntime‚4749  _runtime‚4749ú¤áýë&åM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43937750 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚848314840269824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21637803 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5923127415296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311978 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374226 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95301985 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9725133 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843979373880081 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5850728719121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚842455162925056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21637530 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5859677344768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚838128645160960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311978 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51364235 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43937750 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5859677344768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚842455145885696 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚455 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚848314481884160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842455145885696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51364235 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11912667 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301985 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11912397 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5850388169119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚842455145885696 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚455 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚848314481884160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842455145885696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51364235 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11912667 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301985 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11912397 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5850388169119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311978 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374226 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95301985 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9725133 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843979373880081 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5850728719121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚842455162925056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21637530 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5859677344768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚838128645160960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311978 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51364235 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43937750 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5859677344768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21824466 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845307668082688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975211008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51809635 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5909360212480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5909360212480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12013828 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851207644645633 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809635 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9810641 trainer/global_step‚5599 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚458 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845307651043328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9810638 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51819612 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855578975211008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187531883931648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333589504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚4789.408114054 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12014093 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21824734 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 % _timestamp‚1.7843622973326578e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44311720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21824466 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845307668082688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975211008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51809635 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5909360212480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5909360212480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12013828 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851207644645633 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809635 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9810641 trainer/global_step‚5599 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚458 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845307651043328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9810638 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51819612 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855578975211008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187531883931648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333589504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚4789.408114054 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12014093 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21824734 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 % _timestamp‚1.7843622973326578e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44311720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚849817197367296 % _timestamp‚1.7843622979520102e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587253248 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193539522546176 trainer/global_step‚5599 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5910031500288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5901345206029 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148030 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372846592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51818687 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12015837 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51818687 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚849817214406656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5910372846592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9812383 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚851355523960589 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96138037 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44319352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21828495 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187565294077952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚845454178754560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44319352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚849817214406656 \HDeviceStatsMonitor.on_train_CÈ‹¸” batch_end/inactive_split_bytes.all.allocated‚193539631825408 _step‚459 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12016109 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138037 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855727587253248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51828678 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21828220 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187565312690176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚849817197367296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148030 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚845454161715200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51828678 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855727228867584&ÍÌÖñ& ëM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5910031500288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚849817197367296 % _timestamp‚1.7843622979520102e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587253248 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193539522546176 trainer/global_step‚5599 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5910031500288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5901345206029 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148030 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372846592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51818687 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12015837 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51818687 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚849817214406656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5910372846592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9812383 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚851355523960589 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96138037 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44319352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21828495 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187565294077952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚845454178754560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44319352 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚849817214406656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193539631825408 _step‚459 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12016109 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138037 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855727587253248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51828678 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21828220 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187565312690176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚849817197367296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148030 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚845454161715200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51828678 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855727228867584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5974228468224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002Ë®¯Î82¡- _wandbruntime‚4790  _runtime‚4790 †8¢ 0loss/FRt1/assignment_loss‚1.7903300333023071 2loss/SRqqt2/assignment_loss‚0.5190548363211059 1loss/SRqqt1/detection_loss‚0.9087900309600191 /loss/FBt1/detection_loss‚0.6550521232586887 epoch‚13 _runtime‚4790.028674942 2loss/SRqqt1/assignment_loss‚0.6087102771912526 1loss/FBt1/assignment_loss‚8.3108267125352e-05 /loss/FRt1/detection_loss‚0.5870418960446113 /loss/FRt2/detection_loss‚1.5809156778221949 /loss/FBt2/detection_loss‚0.6804669567443911 1loss/SRqqt2/detection_loss‚0.6547986204398799 $ _timestamp‚1.784362297953517e+09 0loss/FRt2/assignment_loss‚0.7619264671206474 2loss/FBt2/assignment_loss‚0.010676655909256042 $loss/total_loss‚8.75784668338148 trainer/global_step‚5599 _step‚460R6"º £† 1loss/SRqqt2/detection_loss‚0.6547986204398799 $ _timestamp‚1.784362297953517e+09 0loss/FRt2/assignment_loss‚0.7619264671206474 2loss/FBt2/assignment_loss‚0.010676655909256042 $loss/total_loss‚8.75784668338148 trainer/global_step‚5599 _step‚460 0loss/FRt1/assignment_loss‚1.7903300333023071 2loss/SRqqt2/assignment_loss‚0.5190548363211059 1loss/SRqqt1/detection_loss‚0.9087900309600191 /loss/FBt1/detection_loss‚0.6550521232586887 epoch‚13 _runtime‚4790.028674942 2loss/SRqqt1/assignment_loss‚0.6087102771912526 1loss/FBt1/assignment_loss‚8.3108267125352e-05 /loss/FRt1/detection_loss‚0.5870418960446113 /loss/FRt2/detection_loss‚1.5809156778221949 /loss/FBt2/detection_loss‚0.6804669567443911Ìò¤‚(:ç ÃêìÒ“÷×¼ disk.drbd2165.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.743946075439453# disk.dm-0.out‚12464.93994140625 disk.rbd0.out‚0& disk.nvme0n1.out‚12464.93994140625 disk.sda.in‚ 0.00390625 proc.cpu.threads‚17 disk.dm-1.out‚ 286.12890625 disk.nvme0n1.in‚ 7163.078125 disk.sda.out‚ 286.12890625 cpu‚26.170817724129318 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625 network.recv‚ 1.286221e+06 disk.sda3.out‚ 286.12890625! proc.memory.rssMB‚ 2743.328125 network.sent‚ 8.718646e+06) disk./.usagePercent‚39.52644043765574 disk.sda2.in‚0 disk.rbd1.out‚96.20166015625$ memory_percent‚97.03130722045898& proc.memory.availableMB‚ 486.390625 disk.dm-0.in‚ 7163.078125 disk.rbd1.in‚ 0.01953125! disk.nvme0c0n1.in‚ 7163.078125( disk.nvme0c0n1.out‚12464.93994140625 disk.sda3.in‚ 0.00390625# disk./.usageGB‚588.972526550293.ËÒ¥‚(:… ÃêìÒŸŠ£½ gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚107.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ^ˆ 2¦- _wandbruntime‚4808  _runtime‚4808*Ñæ†=§€{ 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18680787086486816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953576326370239 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.692420244216919 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9532185792922974 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45959118008613586 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6895034313201904 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5965766310691833 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8912448883056641 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6796612739562988 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18574349582195282 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9541059136390686 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6659762859344482 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.680292546749115 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20741216838359833 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8919768333435059 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9861757159233093 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3804178833961487 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7569168210029602 'particle/f_score‚0.6408820748329163 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8239796161651611 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9090702533721924 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5838500261306763 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_5_of_5‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.626946210861206 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48141512274742126 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6696118116378784 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6525031328201294 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043837279081344604 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03454127162694931 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9058160185813904 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1867036521434784 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8950679302215576 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚8.226464447634987 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6655188798904419 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3049810826778412 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8862054944038391 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20031997561454773 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1946716010570526 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9076340794563293 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07273323088884354 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5788823962211609 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13882452249526978 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99530029296875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031069831922650337 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981736540794373 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14448341727256775 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 +particle/sensitivity‚0.6339513063430786 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978293180465698 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7919436097145081 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3919168710708618 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945455193519592 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032520171254873276 particle/accuracy_3_of_4‚1 7validation_average_jet_accuracy‚0.36284881830215454 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.668501079082489 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14768308401107788 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (jet/accuracy_1_of_4‚0.96824711561203 trainer/global_step‚5599 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6732810139656067 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29036372900009155 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.049533743411302567 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8555216789245605 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7308534979820251 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7324383854866028 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03217069059610367 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.736740231513977 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8422542810440063 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12866319715976715 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15525195002555847 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9919615983963013 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7312403917312622 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4631974697113037 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7224122285842896 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7592335939407349 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8555216789245605 particle/accuracy_3_of_3‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863179326057434 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.635988175868988 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17608287930488586 *jet/accuracy_2_of_2‚0.1460905224084854 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6780023574829102 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033077068626880646 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21923616528511047 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9911101460456848 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20357806980609894 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7642581462860107 *jet/accuracy_2_of_4‚0.8124232292175293 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7236458659172058 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08591166138648987 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17247380316257477 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48074278235435486 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.026119248941540718 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0334261953830719 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7348280549049377 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487890481948853 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 (particle/accuracy‚0.8474275469779968 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25946369767189026 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4468604624271393 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20670072734355927 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026148399338126183 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981736540794373 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7843623165417917e+09 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1948738843202591 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19354230165481567 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_1_of_1‚0.31778815388679504 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19989868998527527 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_3‚0.9210671186447144 ,jet/accuracy_4_of_4‚0.013899499550461769 epoch‚13 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6187031865119934 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15874695777893066 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20581942796707153 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6436899900436401 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19354230165481567 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38017067313194275 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17194560170173645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_2‚0.7875924110412598 _runtime‚ 4808.61773576 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213458299636841 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01771644502878189 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20944663882255554 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962416887283325 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943559765815735 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2683592438697815 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.641743004322052 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6464604735374451 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8995463848114014 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15916666388511658 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.737354576587677 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7311425805091858 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22862860560417175 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026148399338126183 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚461 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5973221659660339 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7950299382209778 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5670833587646484 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998511791229248 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6464604735374451 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7226458787918091 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4744267165660858 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_2_of_3‚0.5573441386222839 ,ÌÙ¶¨…{ 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03217069059610367 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.736740231513977 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8422542810440063 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12866319715976715 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15525195002555847 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9919615983963013 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7312403917312622 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4631974697113037 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7224122285842896 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7592335939407349 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747;û†Õ9495174408 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8555216789245605 particle/accuracy_3_of_3‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863179326057434 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.635988175868988 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17608287930488586 *jet/accuracy_2_of_2‚0.1460905224084854 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6780023574829102 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033077068626880646 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21923616528511047 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9911101460456848 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20357806980609894 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7642581462860107 *jet/accuracy_2_of_4‚0.8124232292175293 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7236458659172058 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08591166138648987 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17247380316257477 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48074278235435486 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.026119248941540718 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0334261953830719 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7348280549049377 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487890481948853 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 (particle/accuracy‚0.8474275469779968 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25946369767189026 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4468604624271393 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20670072734355927 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026148399338126183 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981736540794373 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7843623165417917e+09 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1948738843202591 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19354230165481567 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_1_of_1‚0.31778815388679504 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19989868998527527 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_3‚0.9210671186447144 ,jet/accuracy_4_of_4‚0.013899499550461769 epoch‚13 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6187031865119934 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15874695777893066 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20581942796707153 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6436899900436401 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19354230165481567 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38017067313194275 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17194560170173645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_2‚0.7875924110412598 _runtime‚ 4808.61773576 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213458299636841 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01771644502878189 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20944663882255554 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962416887283325 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943559765815735 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2683592438697815 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.641743004322052 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6464604735374451 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8995463848114014 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15916666388511658 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.737354576587677 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7311425805091858 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22862860560417175 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026148399338126183 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚461 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5973221659660339 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7950299382209778 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5670833587646484 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998511791229248 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6464604735374451 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7226458787918091 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4744267165660858 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_2_of_3‚0.5573441386222839 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18680787086486816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953576326370239 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.692420244216919 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9532185792922974 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45959118008613586 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6895034313201904 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5965766310691833 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8912448883056641 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6796612739562988 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18574349582195282 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9541059136390686 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6659762859344482 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.680292546749115 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20741216838359833 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_4_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8919768333435059 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9861757159233093 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3804178833961487 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7569168210029602 'particle/f_score‚0.6408820748329163 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8239796161651611 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9090702533721924 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5838500261306763 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_5_of_5‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.626946210861206 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48141512274742126 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6696118116378784 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6525031328201294 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043837279081344604 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03454127162694931 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9058160185813904 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1867036521434784 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8950679302215576 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚8.226464447634987 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6655188798904419 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3049810826778412 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8862054944038391 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20031997561454773 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1946716010570526 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9076340794563293 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07273323088884354 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5788823962211609 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13882452249526978 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99530029296875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031069831922650337 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981736540794373 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14448341727256775 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 +particle/sensitivity‚0.6339513063430786 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978293180465698 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7919436097145081 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3919168710708618 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945455193519592 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032520171254873276 particle/accuracy_3_of_4‚1 7validation_average_jet_accuracy‚0.36284881830215454 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.668501079082489 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14768308401107788 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (jet/accuracy_1_of_4‚0.96824711561203 trainer/global_step‚5599 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6732810139656067 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29036372900009155 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.049533743411302567 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8555216789245605 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7308534979820251 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7324383854866028 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 jet/accuracy_1_of_5‚1Í÷‘g©‚(:’ ÒêìÒ’øéº gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04.֤媂(:Ú ÒêìÒ¢„¤¼ network.sent‚ 8.781583e+06$ disk./.usageGB‚588.9726486206055* proc.memory.percent‚16.770148277282715 disk.sda2.in‚0 disk.dm-0.in‚ 7163.078125 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 7163.078125 disk.nvme0n1.in‚ 7163.078125 disk.sda3.out‚ 286.828125# memory_percent‚97.0590353012085* disk./.usagePercent‚39.526448629896834 disk.rbd1.in‚ 0.01953125 disk.sda1.in‚0( proc.memory.availableMB‚ 481.84765625 network.recv‚ 1.290458e+06 disk.dm-0.out‚ 12471.1484375 disk.dm-1.out‚ 286.828125 disk.rbd1.out‚96.20166015625 disk.sda.out‚ 286.828125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625$ disk.nvme0c0n1.out‚ 12471.1484375" disk.nvme0n1.out‚ 12471.1484375 disk.sda2.out‚0# proc.memory.rssMB‚ 2747.62109375 disk.sda.in‚ 0.00390625 cpu‚26.175355508037956 proc.cpu.threads‚17_.«‚(:’ áêìÒ¸µ´º gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B7 Jö¬‚(:ë áêìÒ­¹É¼ disk.nvme0n1.in‚ 7163.078125 disk.sda.out‚ 287.18359375 cpu‚26.174655842546066 disk.dm-0.in‚ 7163.078125 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0( disk.nvme0c0n1.out‚12473.85498046875 disk.sda.in‚ 0.00390625 disk.sda1.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 481.83984375# disk.dm-0.out‚12473.85498046875 network.sent‚ 8.83875e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625! disk.nvme0c0n1.in‚ 7163.078125$ memory_percent‚97.05908298492432 network.recv‚ 1.294808e+06 disk.sda2.out‚0* proc.memory.percent‚16.770148277282715# disk./.usageGB‚588.972110748291* disk./.usagePercent‚39.526412532834506 disk.rbd1.in‚ 0.01953125 disk.rbd1.out‚96.32666015625& disk.nvme0n1.out‚12473.85498046875 disk.sda3.out‚ 287.18359375 disk.sda2.in‚0# proc.memory.rssMB‚ 2747.62109375 proc.cpu.threads‚17 disk.dm-1.out‚ 287.18359375 disk.rbd0.out‚0ĉ‡ ž­‚(:“ ðêìÒœšƒ¼ gpu.0.gpu‚31.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Ó‰iö®‚(:ë ðêìÒ’þæ¼ network.sent‚ 8.843737e+06* disk./.usagePercent‚39.526415860932445 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 7163.078125 disk.nvme0n1.in‚ 7163.078125 disk.drbd2165.in‚0 disk.dm-1.out‚ 287.41796875 disk.rbd1.out‚96.32666015625 disk.dm-1.in‚ 0.00390625# proc.memory.rssMB‚ 2747.62109375 network.recv‚ 1.297899e+06& disk.nvme0n1.out‚12478.44580078125 disk.sda3.out‚ 287.41796875 disk.sda2.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12478.44580078125 disk.sda.in‚ 0.00390625 disk.sda1.out‚0 disk.rbd1.in‚ 0.01953125 disk.dm-0.in‚ 7163.078125* proc.memory.percent‚16.770148277282715 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0# disk.dm-0.out‚12478.44580078125$ memory_percent‚97.05643653869629' proc.memory.availableMB‚ 482.2734375 disk.drbd2165.out‚0 cpu‚26.17421804820641 proc.cpu.threads‚17$ disk./.usageGB‚588.9721603393555 disk.sda.out‚ 287.41796875 disk.sda1.in‚0»Ý¢U2¯- _wandbruntime‚4848  _runtime‚4848pb É°Ã ' lr-AdamW/pg1‚0.00036245618685214106 ' lr-AdamW/pg2‚0.00036245618685214106 trainer/global_step‚5649 % _timestamp‚1.7843623569168158e+09 _runtime‚4848.992248834 _step‚462aºñõÎ±È _step‚462 ' lr-AdamW/pg1‚0.00036245618685214106 ' lr-AdamW/pg2‚0.00036245618685214106 trainer/global_step‚5649 % _timestamp‚1.7843623569168158e+09 _runtime‚4848.992248834ÎDõQ³2²- _wandbruntime‚4848  _runtime‚4848_£ ºâ'³ÜO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22047999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5977788919808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44813984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6042571796992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158372 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813986 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22048247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚858961591508992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189219233957888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52344386 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5977447580672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚860510177665450 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148393 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9900411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4848.996100975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9900408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189219215345664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚864939380428800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195261787142656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022050304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚858961574469632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚860510535254700 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52334409 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195261898526208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5977788919808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022050304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚858961591508992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52344386 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813986ÆõE¿J ´áO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022050304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚858961591508992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52344386 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813986 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22047999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5977788919808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44813984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6042571796992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158372 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813986 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22048247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADevicy®S†eStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚858961591508992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189219233957888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52344386 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5977447580672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚860510177665450 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148393 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9900411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4848.996100975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9900408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189219215345664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚864939380428800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195261787142656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022050304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚858961574469632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚860510535254700 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52334409 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195261898526208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5977788919808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888Ïo?[»2µ- _wandbruntime‚4849  _runtime‚4849yîZªë&¶åM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175070 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859108815499264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189252662716416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969681418166 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚859108832538624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5978801553920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44821616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9902153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189252644104192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97165077 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44821618 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195296350946304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5969340868164 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865087275706880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22051700 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52353452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚854688358522880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865087634092544 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6043688229888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859108832538624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175070 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784362357535578e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚865087634092544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978460207616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859108815499264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚5649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5978460207616 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52343461 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97165077 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44821618 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195296350946304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5969340868164 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865087275706880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22051700 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52353452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚854688358522880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865087634092544 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6043688229888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859108832538624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175070 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784362357535578e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚865087634092544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978460207616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859108815499264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚5649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5978460207616 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52343461 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52788861 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12251219 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984445 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45195586 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769033728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22238880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196984646744064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚866323642990592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚872352127412224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872351769033728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6093890477056 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872352127412224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚5699 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784362397649764e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚866323642990592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196984543749120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190890671884288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9987661 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚4889.725431097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143082496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798838 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867886685335208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚861867381444608 _step‚467 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867886327745958 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6028484421632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45195586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0AFÚæ'ÄàO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6019286851240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22238606 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6028484421632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872352127412224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚5699 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784362397649764e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚866323642990592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196984543749120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190890671884288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9987661 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚4889.725431097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143082496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97994424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798838 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867886685335208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚861867381444608 _step‚467 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867886327745958 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6028484421632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45195586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6093974859776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52798838 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚866323625951232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12250948 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861867398483968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚866323625951232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52788861 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12251219 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984445 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45195586 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769033728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22238880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196984646744064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚866323642990592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚872352127412224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872351769033728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6093890477056 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22242448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197018998287872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6019957355072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚866470884020224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843623984524882e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001129 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚872500381075968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45203216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22242724 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011122 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060914 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52807904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚866470866980864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6094916257280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360ÿ20ÉëM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚868033849471552 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190924100642816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190924082030592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45203218 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _step‚468 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52797913 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6029497055744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52807904 H@DeviceStatsMonitor.on_train_batch_end/inactive_spÂ9¸î!lit.all.current‚276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚872500381075968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45203216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22242724 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011122 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060914 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52807904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚866470866980864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6094916257280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22242448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197018998287872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6019957355072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚866470884020224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843623984524882e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001129 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6145116774912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22430012 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22429729 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 _runtime‚4930.731305185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45577184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6079179923456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53243313 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6145213740544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516017152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830476 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45577186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77}±±å"ÖâO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879764874395648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98820497 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6078838584320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6145116774912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22430012 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22429729 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 _runtime‚4930.731305185 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45577184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6079179923456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53243313 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6145213740544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516017152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830476 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45577186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873685694472192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12355101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6079179923456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10074911 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12354821 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚869192915038208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6069903338148 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10074908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198707207973376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875262835415716 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45577184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53253290 ^JDeviceStatsMonitor.on_train_bùhxiatch_start/inactive_split_bytes.all.allocated‚198707323551232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879764516017152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6078838584320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0×lâ´82×- _wandbruntime‚4931  _runtime‚4931S8åí&ØçM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45584818 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53252365 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875409999552060 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚869339425710080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45584818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6146150708224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45584816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769673728 _runtime‚4931.346058771 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198741670665216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851211264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12356897 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12357158 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875410357141422 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22433814 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879913128059392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53262356 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873832935501824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚5749 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847174 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879913128059392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192557568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784362439270453e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10076653 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6079851211264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45584816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198741771555840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚873832918462464 _step‚472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192595538569216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22433550 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847174 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53252365 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53262356 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45584818 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53252365 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875409999552060 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚869339425710080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45584818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6146150708224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45584816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769673728 _runtime‚4931.346058771 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198741670665216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851211264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12356897 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12357158 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875410357141422 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22433814 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879913128059392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53262356 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873832935501824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚5749 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847174 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879913128059392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192557568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784362439270453e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10076653 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6079851211264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45584816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198741771555840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚873832918462464 _step‚472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192595538569216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22433550 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847174 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53252365 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53262356 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887177263000576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚881047745953792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53707742 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6120179275166 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53697765 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22619922 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200429710778880 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚881047745953792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6129875425280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚5799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876518448631808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6129534086144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6196274425344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚882638985496224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843624796215699e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6129875425280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53707742 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263000576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚887177621379072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958786 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200429822162432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534086144 _step‚475 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10162158 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194233529124864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876518465671168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚ 4971.69715062 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚881047728914432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6196181654016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22620194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚881047728914432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496‰%öè'èâO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚887177621379072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958786 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200429822162432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534086144 _step‚475 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10162158 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194233529124864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876518465671168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚ 4971.69715062 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚881047728914432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6196181654016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22620194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚881047728914432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45958786 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12458033 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12457764 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887177263000576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚881047745953792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53707742 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6120179275166 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53697765 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22619922 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200429710778880 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚881047745953792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6129875425280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚5799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚479 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876518448631808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6129534086144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6196274425344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚882638985496224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843624796215699e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6129875425280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53707742 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263000576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983Û“Þå>2é- _wandbruntime‚4972  _runtime‚4972¶ƒî&êèM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882786149632568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚876664976343040 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45966418 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚881194969944064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚881194986983424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683226 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12459765 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4972.308650936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6197194550272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45966416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194266976495616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683226 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22623668 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚881194986983424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6130888059392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1™]µ1ëíM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882786507221930 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45966418 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚881194969944064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚881194986983424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683226 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12459765 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4972.308650936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6197194550272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45966416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194266976495616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683226 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22623668 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚881194986983424 \JDeviceStatsMonitor.on_train_batch_end/allocatedä›Ú× _bytes.small_pool.allocated‚ 6130888059392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882786149632568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚876664976343040 EÍ“ñ‚(:ˆ †ììÒ‹ë¹» gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€)ƒïò‚(:ä †ììÒÌÒ¾½$ memory_percent‚97.06196784973145 network.sent‚ 9.162307e+06! disk.nvme0c0n1.in‚ 7435.609375! disk.dm-0.out‚12526.982421875' proc.memory.availableMB‚ 481.3671875$ disk./.usageGB‚588.9560508728027 disk.sda.in‚ 0.00390625) disk./.usagePercent‚39.52533474111535& disk.nvme0c0n1.out‚12526.982421875 disk.sda.out‚ 291.98828125 disk.rbd1.out‚96.74853515625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 7435.609375 disk.rbd1.in‚ 0.01953125* proc.memory.percent‚16.770148277282715 proc.cpu.threads‚17 network.recv‚ 1.337256e+06 disk.nvme0n1.in‚ 7435.609375 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚12526.982421875 disk.dm-1.out‚ 291.98828125# proc.memory.rssMB‚ 2747.62109375 disk.sda1.out‚0 cpu‚26.16985101712839 disk.rbd0.out‚0 disk.sda3.out‚ 291.98828125 disk.dm-1.in‚ 0.00390625õŒNžó‚(:“ •ììÒÕܺ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚139.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.879199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÛ%Ãèô‚(:Ý •ììÒ°í½ disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 7435.61328125* proc.memory.percent‚16.770148277282715( proc.memory.availableMB‚ 481.11328125 disk.dm-1.out‚292.8125 disk.rbd1.in‚ 0.01953125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 7435.61328125 disk.sda3.in‚ 0.00390625 cpu‚26.16924829880435( disk./.usagePercent‚39.5253306449948 disk.rbd1.out‚96.81103515625! disk.nvme0n1.in‚ 7435.61328125 disk.sda2.in‚0 disk.drbd2165.out‚0! disk.dm-0.out‚12531.138671875 disk.rbd0.out‚0$ disk.nvme0n1.out‚12531.138671875 disk.sda.out‚292.8125# memory_percent‚97.0635175704956 proc.cpu.threads‚17 network.recv‚ 1.341094e+06 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625& disk.nvme0c0n1.out‚12531.138671875 disk.sda.in‚ 0.00390625$ disk./.usageGB‚588.9559898376465 disk.sda2.out‚0 disk.sda3.out‚292.8125# proc.memory.rssMB‚ 2747.62109375 network.sent‚ 9.167426e+06Á\u2õ- _wandbruntime‚5012  _runtime‚50125$ݬÉöà ' lr-AdamW/pg2‚0.00035718442378963487 trainer/global_step‚5849 % _timestamp‚1.7843625205897894e+09 _runtime‚5012.668204311 _step‚478 ' lr-AdamW/pg1‚0.00035718442378963487ÖsÑ/Î÷È ' lr-AdamW/pg1‚0.00035718442378963487 ' lr-AdamW/pg2‚0.00035718442378963487 trainer/global_step‚5849 % _timestamp‚1.7843625205897894e+09 _runtime‚5012.668204311 _step‚478ÞC{ 2ø- _wandbruntime‚5012  _runtime‚5012-$›‡è'ùâO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890014777987482 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105354240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202152110101504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195904967051264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894590368362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6171136311964 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6247143050240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10249411 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚894590368362496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46340386 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123966464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6180570927104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚894590009984000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54152217 trainer/global_step‚5849 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22810621 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚ 5012.66974515 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009984000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46340384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888409780396032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10249408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229587968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22810352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570927104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229587968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12560944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6247248404480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54162194 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576732 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6170795762074 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888409797435392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202152234067968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚883843982225408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54152217 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54152217 trainer/global_step‚5849 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22810621 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚ 5012.66974515 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009984000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46340384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888409780396032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10249408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229587968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22810352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570927104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229587968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12560944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6247248404480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54162194 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576732 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6170795762074 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888409797435392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202152234067968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚883843982225408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54152217 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_trainb; _batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54161269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519278 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46348016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10251153 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263640576 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 % _timestamp‚1.7843625212075155e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509285 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22814372 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22814107 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚5013.283147702 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181242214912¦Ð÷&ýñM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263640576 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 % _timestamp‚1.7843625212075155e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509285 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22814372 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22814107 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚5013.283147702 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181242214912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54171260 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6181242214912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12563216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚883990509936640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894738622026240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54161269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54161269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519278 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46348016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10251153 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479à{žBŒ2þ- _wandbruntime‚5013  _runtime‚5013d²? ÿƒ 2loss/FBt1/assignment_loss‚0.038773241926128674 1loss/FBt2/assignment_loss‚0.08802260432033321 /loss/FRt1/detection_loss‚0.5829278149501237 1loss/SRqqt2/detection_loss‚0.6106814337684618 /loss/FBt1/detection_loss‚0.6210803900845349 /loss/FBt2/detection_loss‚0.5422597932271086 %loss/total_loss‚8.583553460486025 0loss/FRt2/assignment_loss‚0.7598473390807277 % _timestamp‚1.7843625212092469e+09 _step‚481 0loss/FRt1/assignment_loss‚1.7454192386958784 2loss/SRqqt2/assignment_loss‚0.5608775820198395 trainer/global_step‚5849 /loss/FRt2/detection_loss‚1.5661323301534613 1loss/SRqqt1/detection_loss‚0.9641130456037706 epoch‚14 _runtime‚5013.284315747 2loss/SRqqt1/assignment_loss‚0.5034186466556562åy¾€ˆ 0loss/FRt1/assignment_loss‚1.7454192386958784 2loss/SRqqt2/assignment_loss‚0.5608775820198395 trainer/global_step‚5849 /loss/FRt2/detection_loss‚1.5661323301534613 1loss/SRqqt1/detection_loss‚0.9641130456037706 epoch‚14 _runtime‚5013.284315747 2loss/SRqqt1/assignment_loss‚0.5034186466556562 2loss/FBt1/assignment_loss‚0.038773241926128674 1loss/FBt2/assignment_loss‚0.08802260432033321 /loss/FRt1/detection_loss‚0.5829278149501237 1loss/SRqqt2/detection_loss‚0.6106814337684618 /loss/FBt1/detection_loss‚0.6210803900845349 /loss/FBt2/detection_loss‚0.5422597932271086 %loss/total_loss‚8.583553460486025 0loss/FRt2/assignment_loss‚0.7598473390807277 % _timestamp‚1.7843625212092469e+09 _step‚481á§ü¦ô›‚(: ¤ììÒ ¬â¼ gpu.0.gpu‚3.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.27839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò?Mñ‚‚(:æ ¤ììÒçàý½$ disk./.usageGB‚588.9639511108398 disk.drbd2165.out‚0 disk.dm-0.in‚ 7435.8671875# proc.memory.rssMB‚ 2747.62109375 cpu‚26.168786979283688 network.recv‚ 1.344098e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 7435.8671875 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚12537.685546875 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0n1.out‚12537.685546875( proc.memory.availableMB‚ 481.79296875* proc.memory.percent‚16.770148277282715 disk.rbd1.in‚ 0.01953125" disk.nvme0c0n1.in‚ 7435.8671875 disk.dm-1.out‚ 293.4609375 disk.sda.out‚ 293.4609375) disk./.usagePercent‚39.52586493271876 disk.sda1.in‚0! disk.dm-0.out‚12537.685546875 proc.cpu.threads‚17 network.sent‚ 9.172413e+06 disk.rbd1.out‚96.81103515625 disk.sda3.out‚ 293.4609375 disk.sda2.out‚0$ memory_percent‚97.05936908721924òöŽ“ƒ‚(:ˆ ³ììÒ©Ãܺ gpu.0.gpu‚92.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚247.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?±ê”ñ„‚(:æ ³ììÒ°ÆŸ½ disk.nvme0n1.in‚ 7517.265625 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 cpu‚26.168245574258705 disk.sda2.out‚0 disk.dm-0.in‚ 7517.265625& proc.memory.availableMB‚ 481.828125 network.recv‚ 1.348745e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.00390625* proc.memory.percent‚16.770148277282715% disk.nvme0n1.out‚12556.6708984375 disk.sda.in‚ 0.00390625# proc.memory.rssMB‚ 2747.62109375$ memory_percent‚97.05915451049805" disk./.usageGB‚588.97021484375' disk.nvme0c0n1.out‚12556.6708984375 disk.sda3.out‚ 293.81640625 disk.sda1.out‚0 disk.dm-1.out‚ 293.81640625 disk.rbd1.out‚96.88916015625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 293.81640625 network.sent‚ 9.235296e+06* disk./.usagePercent‚39.526285297089984 disk.rbd1.in‚ 0.01953125! disk.nvme0c0n1.in‚ 7517.265625" disk.dm-0.out‚12556.6708984375ƒ°ÅQ2…- _wandbruntime‚5053  _runtime‚5053m>ŒÈ†Â ' lr-AdamW/pg2‚0.00035584384074347967 trainer/global_step‚5899 $ _timestamp‚1.784362561349817e+09 _runtime‚5053.432171603 _step‚482 ' lr-AdamW/pg1‚0.00035584384074347967ÜR#êÍ‡Ç ' lr-AdamW/pg1‚0.00035584384074347967 ' lr-AdamW/pg2‚0.00035584384074347967 trainer/global_step‚5899 $ _timestamp‚1.784362561349817e+09 _runtime‚5053.432171603 _step‚482âD‰/2ˆ- _wandbruntime‚5053  _runtime‚5053ˇFè'‰âO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248982 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46721984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54606669 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266428928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197576423589888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756967424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚891169515819008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328653 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54606669 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115345920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23001175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚897390928067990 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115345920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203874647639040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚5053.432282994 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902002756967424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231266428928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6298341724160 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6298242661376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101338632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚5899 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6230925089792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54616646 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897391285657240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46721986 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12664514 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚031°GZŠçO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197576423589888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756967424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚891169515819008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 1013286‹¥®“%53 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54606669 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115345920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23001175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚897390928067990 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115345920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203874647639040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚5053.432282994 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902002756967424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231266428928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6298341724160 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6298242661376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101338632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚5899 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6230925089792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54616646 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897391285657240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46721986 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12664514 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248982 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46721984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54606669 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266428928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872ãE02‹- _wandbruntime‚5054  _runtime‚5054Ó;!—ð&ŒêM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚5054.043986109 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895919089946624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚895919072907264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚902151010624000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6299268475392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897538807382946 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚891316043530240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6231937716736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54615721 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902151369009664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚484 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46729616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚902151369009664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203909211490816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚895919072907264 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203909102211584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12666584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12666315 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46729618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚895919089946624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197609833736192 trainer/global_step‚5899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197609852348416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897538449793584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6222763852706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6299359142400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23004718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222423302704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46729618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚895919089946624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197609833736192 trainer/global_step‚5899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197609852348416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897538449793584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6222763852706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6299359142400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23004718 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222423302704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚891316043530240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6231937716736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54615721 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902151369009664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚484 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46729616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚902151369009664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203909211490816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚895919072907264 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203909102211584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12666584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12666315äÙ™›Å2Ž- _wandbruntime‚5054  _runtime‚5054$¶; 1loss/SRqqt1/detection_loss‚0.8328227696816126 epoch‚14 0loss/FRt2/assignment_loss‚0.7046148446324754 -loss/FRt2/detection_loss‚1.52612087824148 /loss/FBt1/detection_loss‚0.5995403953711502 % _timestamp‚1.7843625619708502e+09 2loss/SRqqt2/assignment_loss‚0.5592346858903895 /loss/FRt1/detection_loss‚0.5263762472864488 /loss/FBt2/detection_loss‚0.6974733673732962 $loss/total_loss‚8.52034082280042 _runtime‚5054.045896235 0loss/FRt1/assignment_loss‚1.7010760346819414 2loss/SRqqt1/assignment_loss‚0.6872981725821107 1loss/FBt1/assignment_loss‚0.03893427619641532 1loss/SRqqt2/detection_loss‚0.6436849744154178 trainer/global_step‚5899 _step‚485 3loss/FBt2/assignment_loss‚0.0031641764476821317 *L † 3loss/FBt2/assignment_loss‚0.0031641764476821317 1loss/SRqqt1/detection_loss‚0.8328227696816126 epoch‚14 0loss/FRt2/assignment_loss‚0.7046148446324754 -loss/FRt2/detection_loss‚1.52612087824148 /loss/FBt1/detection_loss‚0.5995403953711502 % _timestamp‚1.7843625619708502e+09 2loss/SRqqt2/assignment_loss‚0.5592346858903895 /loss/FRt1/detection_loss‚0.5263762472864488 /loss/FBt2/detection_loss‚0.6974733673732962 $loss/total_loss‚8.52034082280042 _runtime‚5054.045896235 0loss/FRt1/assignment_loss‚1.7010760346819414 2loss/SRqqt1/assignment_loss‚0.6872981725821107 1loss/FBt1/assignment_loss‚0.03893427619641532 1loss/SRqqt2/detection_loss‚0.6436849744154178 trainer/global_step‚5899 _step‚485å]¦êå‘‘‚(:† ÂììÒƒŽÑº gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oD‘ªï’‚(:ä ÂììÒã­’½ disk.dm-1.out‚ 294.359375# disk./.usageGB‚588.970100402832 disk.sda1.in‚0 disk.drbd2165.in‚0# disk.dm-0.out‚12564.95654296875 network.recv‚ 1.351975e+06 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 7517.265625( disk.nvme0c0n1.out‚12564.95654296875 cpu‚26.167602158586767 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01953125 disk.sda.in‚ 0.00390625' proc.memory.availableMB‚ 481.2265625) disk./.usagePercent‚39.52627761686396 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 294.359375 proc.cpu.threads‚17 network.sent‚ 9.240281e+06 disk.nvme0n1.in‚ 7517.265625& disk.nvme0n1.out‚12564.95654296875 disk.sda.out‚ 294.359375$ memory_percent‚97.06282615661621# proc.memory.rssMB‚ 2747.62109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚96.88916015625* proc.memory.percent‚16.770148277282715 disk.dm-0.in‚ 7517.265625 disk.rbd0.in‚0 disk.sda3.in‚ 0.00390625E²ÝK“‚(:„ ÑììÒþ·’¼ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.mûk÷DemoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôª‹õ”‚(:ê ÑììÒ–Ðܽ disk.dm-1.out‚ 294.67578125( disk.nvme0c0n1.out‚12569.43896484375& disk.nvme0n1.out‚12569.43896484375* proc.memory.percent‚16.770148277282715$ memory_percent‚97.05936908721924 disk.sda.in‚ 0.00390625 cpu‚26.16744598372496 proc.cpu.threads‚17 network.recv‚ 1.355989e+06$ disk./.usageGB‚588.9699974060059 disk.sda2.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2747.62109375 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚96.95166015625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 294.67578125 disk.sda1.out‚0( proc.memory.availableMB‚ 481.79296875 network.sent‚ 9.30237e+06) disk./.usagePercent‚39.52627070466053 disk.dm-0.in‚ 7517.265625# disk.dm-0.out‚12569.43896484375 disk.rbd0.out‚0 disk.rbd1.in‚ 0.01953125 disk.nvme0n1.in‚ 7517.265625 disk.sda.out‚ 294.67578125 disk.drbd2165.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 7517.265625 disk.sda1.in‚0$ÙI8•‚(:’ àììÒ缺 gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jcô0ô–‚(:é àììÒ߬½ disk.sda2.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 7517.265625( disk.nvme0c0n1.out‚12575.84326171875 disk.nvme0n1.in‚ 7517.265625 disk.drbd2165.out‚0 disk.dm-0.in‚ 7517.265625& disk.nvme0n1.out‚12575.84326171875 disk.sda1.out‚0# disk.dm-0.out‚12575.84326171875 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0& proc.memory.availableMB‚ 481.515625 disk.sda3.in‚ 0.00390625 disk.dm-1.out‚ 295.23828125 proc.cpu.threads‚17 network.sent‚ 9.307351e+06$ disk./.usageGB‚588.9701042175293 disk.drbd2165.in‚0 disk.rbd1.out‚96.95166015625* proc.memory.percent‚16.770148277282715 network.recv‚ 1.359305e+06) disk./.usagePercent‚39.52627787287149 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01953125$ memory_percent‚97.06106185913086 disk.sda3.out‚ 295.23828125# proc.memory.rssMB‚ 2747.62109375 disk.sda.in‚ 0.00390625 disk.sda.out‚ 295.23828125 cpu‚26.16686053773005hE%è2—- _wandbruntime‚5094  _runtime‚5094­§&ǘÁ & lr-AdamW/pg1‚0.0003544943721237577 & lr-AdamW/pg2‚0.0003544943721237577 trainer/global_step‚5949 % _timestamp‚1.7843626022707958e+09 _runtime‚5094.349223372 _step‚486~ mÌ™Æ & lr-AdamW/pg1‚0.0003544943721237577 & lr-AdamW/pg2‚0.0003544943721237577 trainer/global_step‚5949 % _timestamp‚1.7843626022707958e+09 _runtime‚5094.349223372 _step‚486æê±i–2š- _wandbruntime‚5094  _runtime‚5094ò#¾Ãç'›áO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚909415503950848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6349471708160 _runtime‚5094.350251524 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10423911 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55061121 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199247842904064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071098 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12768255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904767078148498 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6281961930752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚909415862329344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6281620591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10423908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903133900398592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47103586 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164705 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199247861516288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚898495066451968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903133900398592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚898495049412608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚487 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47103586 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503950848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737748 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚11691°vÀì'œæO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚898495049412608 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚487 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47103586 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503950848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737748 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚909415503950848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6349471708160 _runtime‚5094.350251524 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10423911 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55061121 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199247842904064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071098 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12768255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904767078148498 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6281961930752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚909415862329344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6281620591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10423908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903133900398592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47103586 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164705 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199247861516288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚898495066451968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903133900398592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976çpŒ2- _wandbruntime‚5094  _runtime‚5094f}æ"žëM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6350395855360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚ 5094.96723106 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191382 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191382 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55070173 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843626028916118e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181389 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199281271662592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12770057 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205631667517952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚903281124388864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚898641577123840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6273039789612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47111218 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10425653 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115993088 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55080164 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23195995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23195710 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904914599874092 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757607424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚5949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55070173 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47111216 KEDeviceStatsMonitor.on_tra÷C‡¿ in_batch_end/reserved_bytes.small_pool.freed‚0 _step‚488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974564864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10425656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ýOìjö&ŸðM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6350486522368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6282633218560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6282633218560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191382 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55070173 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚479 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843626028916118e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181389 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199281271662592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12770057 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205631667517952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚903281124388864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚898641577123840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6273039789612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47111218 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10425653 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115993088 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55080164 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23195995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23195710 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904914599874092 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757607424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚5949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55070173 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47111216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974564864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10425656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6350395855360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚ 5094.96723106 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191382 E““§‚(:ˆ íìÒ‡âϺ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sèÉwù¨‚(:î íìÒî½( disk./.usagePercent‚39.5262712166756" disk.dm-0.out‚12587.0244140625 disk.dm-1.in‚ 0.00390625 disk.rbd0.in‚0 disk.sda3.out‚ 296.76171875 disk.sda1.in‚0 disk.dm-0.in‚ 7600.73046875 proc.cpu.threads‚17' disk.nvme0c0n1.out‚12587.0244140625 disk.dm-1.out‚ 296.76171875 disk.rbd1.out‚97.09228515625 disk.sda2.out‚0$ memory_percent‚97.06127643585205 cpu‚26.165649741304865( proc.memory.availableMB‚ 481.48046875$ disk./.usageGB‚588.9700050354004! disk.nvme0n1.in‚ 7600.73046875% disk.nvme0n1.out‚12587.0244140625 disk.sda.out‚ 296.76171875 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0 network.recv‚ 1.370681e+06 disk.sda2.in‚0# proc.memory.rssMB‚ 2747.62109375 disk.rbd1.in‚ 0.01953125 disk.sda.in‚ 0.00390625 network.sent‚ 9.380336e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.770148277282715 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7600.73046875þA‘2©- _wandbruntime‚5135  _runtime‚5135;¶ûÇƪÀ % _timestamp‚1.7843626430248053e+09 _runtime‚ 5135.10014533 _step‚490 & lr-AdamW/pg1‚0.0003531361046043167 & lr-AdamW/pg2‚0.0003531361046043167 trainer/global_step‚5999XÁL‰Ë«Å _runtime‚ 5135.10014533 _step‚490 & lr-AdamW/pg1‚0.0003531361046043167 & lr-AdamW/pg2‚0.0003531361046043167 trainer/global_step‚5999 % _timestamp‚1.7843626430248053e+09êbعN2¬- _wandbruntime‚5135  _runtime‚5135jìANè'­âO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12871945 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12872218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910495934840832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55515573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910495934840832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23383379 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6332657432576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905820600045568 _step‚491 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916828609312768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6332316093440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47485184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250934272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916828250934272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912143585818256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912143228229006 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916828609312768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905820583006208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23383103 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910495951880192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207319938679808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657432576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910495934840832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55515573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910495934840832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23383379 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6332657432576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905820600045568 _step‚491 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916828609312768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6332316093440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47485184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250934272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916828250934272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912143585818256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912143228229006 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916828609312768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905820583006208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23383103 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910495951880192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207319938679808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657432576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910643175870464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027434 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910643192909824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910643175870464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6333328720384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚905967093678080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843626436377413e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12874046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912291107543962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚912290749954600 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23386949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10512903 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916976862976512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6333328720384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6401568561152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47492816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027434 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6401659228160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55524625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12874300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55534616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200952728201216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200952709588992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916976504590848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55524625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23387206 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888úCŽ÷&±ñM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916976862976512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6333328720384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6401568561152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47492816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027434 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6401659228160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55524625 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12874300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55534616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200952728201216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200952709588992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916976504590848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55524625 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23387206 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910643175870464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027434 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910643192909824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910643175870464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6333328720384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚905967093678080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843626436377413e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12874046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912291107543962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚912290749954600 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23386949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10512903ìé‡Óâ2²- _wandbruntime‚5135  _runtime‚5135ö<õ{³‚ /loss/FBt1/detection_loss‚0.6807089452631772 _step‚493 0loss/FRt1/assignment_loss‚1.7320589226012688 /loss/FRt2/detection_loss‚1.5305973000669622 epoch‚14 trainer/global_step‚5999 0loss/FRt2/assignment_loss‚0.7041316274131275 2loss/SRqqt1/assignment_loss‚0.6391154870591667 2loss/SRqqt2/assignment_loss‚0.5009430756310007 /loss/FRt1/detection_loss‚0.5368846608845785 1loss/SRqqt2/detection_loss‚0.6948699459473187 /loss/FBt2/detection_loss‚0.5459599969674089 %loss/total_loss‚8.683364657325278 1loss/SRqqt1/detection_loss‚1.0287731952637946 % _timestamp‚1.7843626436395426e+09 _runtime‚5135.714606744 1loss/FBt1/assignment_loss‚0.04597343406172172 1loss/FBt2/assignment_loss‚0.04334806616575278ÔqF ´‡ 0loss/FRt2/assignment_loss‚0.7041316274131275 2loss/SRqqt1/assignment_loss‚0.6391154870591667 2loss/SRqqt2/assignment_loss‚0.5009430756310007 /loss/FRt1/detection_loss‚0.5368846608845785 1loss/SRqqt2/detection_loss‚0.6948699459473187 /loss/FBt2/detection_loss‚0.5459599969674089 %loss/total_loss‚8.683364657325278 1loss/SRqqt1/detection_loss‚1.0287731952637946 % _timestamp‚1.7843626436395426e+09 _runtime‚5135.714606744 1loss/FBt1/assignment_loss‚0.04597343406172172 1loss/FBt2/assignment_loss‚0.04334806616575278 /loss/FBt1/detection_loss‚0.6807089452631772 _step‚493 0loss/FRt1/assignment_loss‚1.7320589226012688 /loss/FRt2/detection_loss‚1.5305973000669622 epoch‚14 trainer/global_step‚5999ía ´’µ‚(:‡ œíìÒõŸ©º gpu.0.gpu‚11.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚257.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß–*ð¶‚(:å œíìÒ³»€½) disk./.usagePercent‚39.52627172869067 disk.dm-0.in‚ 7600.73046875! disk.nvme0n1.in‚ 7600.73046875 proc.memory.rssMB‚ 2711.09375# disk.nvme0c0n1.in‚ 7600.73046875 disk.sda3.in‚ 0.00390625$ memory_percent‚96.83811664581299 disk.rbd0.in‚0& disk.nvme0c0n1.out‚12591.529296875$ disk.nvme0n1.out‚12591.529296875 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 297.1015625 disk.drbd2165.in‚0 network.sent‚ 9.385325e+06! disk.dm-0.out‚12591.529296875 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 297.1015625 disk.sda2.in‚0 disk.sda1.in‚0 network.recv‚ 1.374215e+06 disk.dm-1.out‚ 297.1015625 disk.sda2.out‚0* proc.memory.percent‚16.547203063964844( proc.memory.availableMB‚ 518.04296875$ disk./.usageGB‚588.9700126647949 disk.rbd1.out‚97.09228515625 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01953125 disk.sda1.out‚0 cpu‚26.17034403179243å·2·- _wandbruntime‚5154  _runtime‚5154g$s€¸±{ =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16995671391487122 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14681854844093323 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 % _timestamp‚1.7843626626838508e+09 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1811545342206955 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02670932374894619 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6677732467651367 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.593917965888977 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8544974327087402 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6261627674102783 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6681103706359863 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6669175028800964 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9909628033638 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7477096319198608 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.153810054063797 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974368810653687 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6563845872879028 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43112415075302124 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15938258171081543 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6647569537162781 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9975833892822266 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6543792486190796 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6070008873939514 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870489835739136 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7416054010391235 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7718632221221924 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15259146690368652 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_5_of_5‚0 trainer/global_step‚5999 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7088043689727783 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02670932374894619 *jet/accuracy_1_of_4‚0.9734241962432861 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01792789064347744 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.723488986492157 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962145686149597 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954373240470886 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11082237213850021 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15151560306549072 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7023851275444031 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48803025484085083 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6552518010139465 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020743627101182938 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt2FBt/