:W&Bá¾n`k…M 33r5r2wdª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0ùPqC‚‚" vi5ofb9eyhra 33r5r2wdŠÞ 33r5r2wdescheuller-uc-san-diego spatop-sweep"Ÿ _wandb‚{}  batch_size‚2048  dropout‚0.13805027573582035  hidden_dim‚64 ( learning_rate‚0.00016045023811285788  num_embedding_layers‚14Bdpfx_blocks_33r5r2wdbox8or7c7j$dp-spatop-sweepagent-blocks-v2-fb276Š ªæôÒÀÇÑÛš3 (H¨¸hp"3.12.10*0.28.1j linux-x86_646Ÿ¼¬; 33r5r2wd*  _wandb‚ {"runtime":0}  _runtime‚03éj•`ÒÚ -Linux-6.8.0-124-generic-x86_64-with-glibc2.36CPython 3.12.10 ªæôÒààø¥*-of*>/data/spatop/options_files/sweep_fixed_dp/config_33r5r2wd.json*-l* /scratch/logs*-n*dpfx_blocks_33r5r2wd2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb$dp-spatop-sweepagent-blocks-v2-fb276r/opt/conda/bin/pythonº ucn8f9cgbcfxr13bc6bb2m1thxlaoekgæPò2  requirements.txtP×à5W 33r5r2wdZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64ã–j„Y 33r5r2wdZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64ûG­* 33r5r2wdb trainer/global_step2.Z}\  33r5r2wdZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64‹ÂÄ/  33r5r2wdb*"trainer/global_step2~ÄÛ¤K  33r5r2wdj: «æôÒø“ Í*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] \sŠÙÇ ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€€ý/€À¤Ú$ª€Àû”ÌÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-ed01d6ad-b673-30ac-3501-763022903663Ê13.2º ucn8f9cgbcfxr13bc6bb2m1thxlaoekgÜCÁ“  33r5r2wdj  «æôÒèßééî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ èB\ 33r5r2wdjÊ «æôÒ Ÿ°ê·Trainable params: 5.3 M Non-trainable params: 50 Total params: 5.3 M Total estimated model params size (MB): 21.374 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 ö/¤"q 33r5r2wdj` «æôÒÐÐÄêPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). @óËf{ 33r5r2wdjj «æôÒø‡ÊêZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). áhjµq 33r5r2wdj` «æôÒàƒÎêPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ¨ÿÝOt 33r5r2wdjc «æôÒ ¹ÑêSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). «kÍin 33r5r2wdj] «æôÒ¨×ÔêMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). cÚÉr+ 33r5r2wd*™  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"SC5ë 33r5r2wdjÙ «æôÒ ÜéìÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ÔfyVN 33r5r2wdj¼ «æôÒàð÷ì«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance.  xÜ7ë 33r5r2wdjÙ ­æôÒè‚Ú¢È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ›<6ÈP 33r5r2wdj¾ ­æôÒ Æõ¢­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Ð$»ñ›‚(:‘ ºæôÒÊ€ó¾ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚116.451' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.81699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦bñ5*‚(:  ºæôÒ‚¡¯¿$ disk.nvme0n1.out‚182.55029296875 disk.md0.out‚ 0.41015625 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2802.4375* proc.memory.percent‚17.104721069335938 network.recv‚41152 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚520.28076171875 disk.md0.in‚0 disk.dm-1.out‚ 0.478515625& disk.nvme0c0n1.out‚182.55029296875) proc.memory.availableMB‚ 1327.19921875) disk./.usagePercent‚76.53598285346442 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.dm-0.out‚ 701.451171875$ memory_percent‚91.89941883087158 proc.cpu.threads‚17% disk./.usageGB‚1175.0186767578125& disk.nvme1c1n1.out‚520.28076171875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 cpu‚25.535187535549902 network.sent‚32720^¶6ç%‚(:› ÉæôÒáÑÔ¾ disk.nvme1n1.in‚0# disk.nvme1n1.out‚520.7685546875 network.recv‚45346 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚520.7685546875 disk.md0.in‚0 disk.md0.out‚0.6875( proc.memory.availableMB‚ 1326.7578125 network.sent‚41205 disk.rbd0.in‚0% disk.nvme0c0n1.out‚183.3427734375# proc.memory.rssMB‚ 2802.44921875* proc.memory.percent‚17.104792594909668% disk./.usageGB‚1175.0179405212402 disk.dm-1.out‚ 0.529296875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ memory_percent‚91.90211296081543 cpu‚25.546807523720496 proc.cpu.threads‚17) disk./.usagePercent‚76.53593489798016 disk.dm-0.out‚ 702.072265625 disk.nvme0n1.in‚0# disk.nvme0n1.out‚183.3427734375n?‚(:“ ÉæôÒ¬†‡¿ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚52.0 gpu.0.powerWatts‚209.606' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚69.86866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+vjÃ3b/ lr-AdamW/pg1"trainer/global_step2:JP#©Îù3b/ lr-AdamW/pg2"trainer/global_step2:JPÇ ‡€-) _wandbruntime‚33  _runtime‚33‚ÄøOÀ » & lr-AdamW/pg1‚1.965515416882509e-05 & lr-AdamW/pg2‚1.965515416882509e-05 trainer/global_step‚49 % _timestamp‚1.7844928764107883e+09 _runtime‚ 33.012701039 _step‚0ŽôxÂ!½ & lr-AdamW/pg2‚1.965515416882509e-05 trainer/global_step‚49 % _timestamp‚1.7844928764107883e+09 _runtime‚ 33.012701039 _step‚0 & lr-AdamW/pg1‚1.965515416882509e-059èrl"bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP¶ô|i#be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPåXè’d$b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPéH™f%bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPʆÇl&bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP˜×æåj'bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP‹ÍÊ-p(bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPôž ^)bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPßô:3f*bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP;¦¯an+bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPƒ²äi,be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP½0áwl-bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÿüƒi.be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPšË0Žp/bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP™ÓF­_0b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP7?†em1bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPM˧k2bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP²úwc3b_ ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPhJ¨o?bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP ‘m@bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP-ìÁlnAbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP7YiBbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP9}–jCbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP¼*|VgDbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPf°nEbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPÒY>¾jFbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPËwyjGbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP©”ÊefHbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP2RäbIb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPÂ^»%xJbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP‘7vmnKbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPé ›nLbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPF2»KjMbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPR=ÙmNbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPd·°VfObb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP°úAjPbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPJ÷DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP¬Zv_br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPmÛŠàh`bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPฌIaab] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP×´ Éibbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP:":qcbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPHwtdbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPèä³]ebY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPªî“efba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP?­ egba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPIçŽ lhbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP—#÷ófibb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPŒ<ß hjbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPhÂikbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP7¶Ã¸nlbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPëRáohmbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP  ëüpnbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPz°Kpobl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP!0›Ôcpb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPœ®z§XˆbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPD©Qðr‰bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP jŠbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPà/i-l‹bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPí—D¡hŒbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPDTybt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPŠ@4ÄaŽb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPÄ‘kbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP_Ôµïsbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPRêl‘bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPÅï~œo’bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP­2¶Íh“bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP7¾Þo”bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPY¹±8p•bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPD +n–bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP•acLm—bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP€n§l˜bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP”ÈSr™bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP/h{ënšbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPâ›) t›bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPOXÎAuœbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP‹'jí.) _wandbruntime‚33  _runtime‚33é!·óÆžþN LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11950 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 40100217856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 39900698930 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 % _timestamp‚1.7844928764119675e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882639 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385421 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 41684976640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚500076 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚490284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚385421 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚500076 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301696512 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40011737600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12021 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚369431 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚801 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚115148 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3776652512768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117040640 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3772847589442 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2893923577344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2852144491008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415743 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3816664250368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3816769769984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2852238600704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.cuë:û¾rrent‚ 105519616 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚115330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269120000 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚369245 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚254097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211150336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3776652512768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269120000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2893712427008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 41567936000 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3772951413700 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _step‚1 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 _runtime‚ 33.013849831 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385423 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3776669552128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚490284 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360;h«6ˆ'Ÿ‚O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882639 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385421 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 41684976640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚500076 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚490284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚385421 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚500076 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301696512 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40011737600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12021 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚369431 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚801 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚115148 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3776652512768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117040640 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3772847589442 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2893923577344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2852144491008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415743 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3816664250368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3816769769984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2852238600704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚115330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269120000 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚369245 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚254097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211150336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3776652512768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269120000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2893712427008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 41567936000 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3772951413700 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _step‚1 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 _runtime‚ 33.013849831 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385423 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3776669552128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚490284 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11950 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 40100217856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 39900698930 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 % _timestamp‚1.7844928764119675e+09ÊÝNœi bd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP2—q¡bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP ¯èÜl¢bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP=zW‰o£bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP.2eßk¤bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP±{ÈPh¥bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPj{7 e¦b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPâOml§bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP ±Çer¨bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP|°ˆæd©b_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPù9Æn·bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP…˜H¾c¸b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP~Œ¡>p¹bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP¥ª¯mºbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPˆÈ²nj»be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP´&ªmm¼bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPFfRÆm½bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPb[•Ýu¾bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPZ’¬fg¿bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP…ï^ÀbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPªÙdÑgÁbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP—¹/fÂba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP‚Æ)œqÃbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP°ûW¨rÄbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP†!|°`Åb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPÚÚÑQdÆb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPaqÔšbÔb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPóã÷¯oÕbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP{A«kÖbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPcóúj×be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP’Ê jØbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPT¥lÙbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPoKTlÚbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPd ÞCkÛbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP À±DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPçŽÝeÿb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP[HžJf€ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP„Ãf^bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JP³A®´m‚bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP6Ã@·rƒbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPx–lp„bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP‡Û€Èd…b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP‘x÷eŽb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPÍÃJ²fba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP¦òKürbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP˜x†jp‘bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPÑN8i’bd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPƬNB]“bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPŒ"¿a”b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP\ÿ(Øf•ba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP÷Ä_u–bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPj$›n—bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP«2¡ e˜b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPìgãq™bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP…R-hšbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP’„ì.›) _wandbruntime‚33  _runtime‚33Ù’Ò\œˆM WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 40885940736 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3892705159168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3851819218432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3851802179072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚376725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚466 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2950660912640 _runtime‚ 33.415804903 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 40797453312 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 40797453312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚49 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚179 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 42378693632 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3848025433804 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 40885940736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2950869958656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2908282219008 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 42493629952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3807237415904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚500104 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚500104 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3892705159168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 40770978540 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393050 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3807254455264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚376542 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114936320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3851802179072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚183 E=DeviceStatsMonitor.on_train_batch_end/inacti|zõ2 ve_split.all.peak‚801 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 40684193530 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844928768137655e+09 _step‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚893154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚117568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 40770978540 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393050 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3807254455264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚376542 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114936320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3851802179072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚183 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚801 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 40684193530 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844928768137655e+09 _step‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚893154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚117568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3851819218432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3851802179072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚376725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚466 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2950660912640 _runtime‚ 33.415804903 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 40797453312 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 40797453312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚49 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚179 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 42378693632 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3848025433804 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812ÇVJAžb< loss/FRt1/assignment_loss"trainer/global_step2:JPl$&‚CŸb> loss/SRqqt2/assignment_loss"trainer/global_step2:JP㈯m@ b; loss/FRt1/detection_loss"trainer/global_step2:JPs&ôA¡b< loss/FBt1/assignment_loss"trainer/global_step2:JPzrâ7¢b2 loss/total_loss"trainer/global_step2:JP߈TLA£b< loss/FRt2/assignment_loss"trainer/global_step2:JP¾Ñ%ãC¤b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPz89î@¥b; loss/FRt2/detection_loss"trainer/global_step2:JPtÊäïB¦b= loss/SRqqt1/detection_loss"trainer/global_step2:JPæï—%B§b= loss/SRqqt2/detection_loss"trainer/global_step2:JP;ÕS@¨b; loss/FBt1/detection_loss"trainer/global_step2:JP“ÈyŠ@©b; loss/FBt2/detection_loss"trainer/global_step2:JP[…w_-ªb( epoch"trainer/global_step2:JPd.¡ÄA«b< loss/FBt2/assignment_loss"trainer/global_step2:JP³¢Ñz.¬) _wandbruntime‚33  _runtime‚33ºÐ]³ø­ò /loss/FBt1/assignment_loss‚2.221492993204217 %loss/total_loss‚69.87847791378675 /loss/FRt2/assignment_loss‚4.627064823794675 1loss/SRqqt1/assignment_loss‚4.226655328363404 /loss/FRt2/detection_loss‚1.2965910879967661 1loss/SRqqt1/detection_loss‚1.9374671926623896 1loss/SRqqt2/detection_loss‚3.7829081877336033 /loss/FBt1/detection_loss‚18.963273897178862 /loss/FBt2/detection_loss‚19.104309419861863 epoch‚0 /loss/FBt2/assignment_loss‚2.266076867921012 trainer/global_step‚49 % _timestamp‚1.7844928768148232e+09 _runtime‚ 33.416776555 .loss/FRt1/assignment_loss‚5.34234676178226 1loss/SRqqt2/assignment_loss‚4.180160440697794 /loss/FRt1/detection_loss‚1.9301309125899038 _step‚3åŒî]ü®ö /loss/FRt2/assignment_loss‚4.627064823794675 1loss/SRqqt1/assignment_loss‚4.226655328363404 /loss/FRt2/detection_loss‚1.2965910879967661 1loss/SRqqt1/detection_loss‚1.9374671926623896 1loss/SRqqt2/detection_loss‚3.7829081877336033 /loss/FBt1/detection_loss‚18.963273897178862 /loss/FBt2/detection_loss‚19.104309419861863 epoch‚0 /loss/FBt2/assignment_loss‚2.266076867921012 trainer/global_step‚49 % _timestamp‚1.7844928768148232e+09 _runtime‚ 33.416776555 .loss/FRt1/assignment_loss‚5.34234676178226 1loss/SRqqt2/assignment_loss‚4.180160440697794 /loss/FRt1/detection_loss‚1.9301309125899038 _step‚3 /loss/FBt1/assignment_loss‚2.221492993204217 %loss/total_loss‚69.87847791378675ø“$¯‚(:™ ØæôÒžã‹¿ network.sent‚66429$ disk./.usageGB‚1175.078899383545 disk.md0.out‚ 1.1015625% disk.nvme0c0n1.out‚204.0244140625 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚526.31005859375$ memory_percent‚91.90785884857178 proc.memory.rssMB‚2802.5 disk.rbd0.out‚0) disk./.usagePercent‚76.53990551268815 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 726.294921875 disk.dm-1.in‚0 disk.dm-1.out‚ 1.63525390625$ disk.nvme1n1.out‚526.31005859375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0) proc.memory.availableMB‚ 1325.81640625( proc.memory.percent‚17.1051025390625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 network.recv‚51355 disk.rbd0.in‚0# disk.nvme0n1.out‚204.0244140625 cpu‚25.5472294006981754ŽŽiž°‚(:“ ØæôÒ†±Á¿ gpu.0.gpu‚53.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚217.784' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚72.59466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iv[?)±‚(:ž çæôÒ¤òã¾% disk.nvme0c0n1.out‚266.6103515625 disk.nvme0n1.in‚0) disk./.usagePercent‚76.53983917012701& disk.nvme1c1n1.out‚531.35107421875 disk.dm-0.in‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚266.6103515625( proc.memory.availableMB‚ 1325.2421875) proc.memory.percent‚17.10512638092041 proc.cpu.threads‚17$ disk./.usageGB‚1175.077880859375 disk.md0.out‚ 1.28515625 disk.dm-0.out‚ 793.3984375 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 2802.50390625 cpu‚25.55478163197458 network.sent‚99492 network.recv‚55611 disk.nvme1n1.in‚0$ memory_percent‚91.91136360168457 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚531.35107421875 disk.md0.in‚0 disk.dm-1.out‚ 1.74853515625 disk.rbd0.in‚0 disk.rbd0.out‚0»õõ²‚(:’ çæôÒœîá¿ gpu.0.gpu‚35.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚220.489' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.49633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cL¿.³) _wandbruntime‚64  _runtime‚64dô½ trainer/global_step‚99 % _timestamp‚1.7844929074655406e+09 _runtime‚ 64.068151936 _step‚4 ' lr-AdamW/pg1‚3.9711433932932325e-05 ' lr-AdamW/pg2‚3.9711433932932325e-05Ùt¾xǵÁ _runtime‚ 64.068151936 _step‚4 ' lr-AdamW/pg1‚3.9711433932932325e-05 ' lr-AdamW/pg2‚3.9711433932932325e-05 trainer/global_step‚99 % _timestamp‚1.7844929074655406e+09»àÿ.¶) _wandbruntime‚64  _runtime‚64V97kˆ'·‚O G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚803 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _runtime‚ 64.068219956 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚732876 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7526612389850 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 79139229738 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7534135827968 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚506897 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7613603025408 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 82320365056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚983136 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚506901 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7844929074662905e+09 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚766871 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 79361677824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚766871 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚733048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7613497505792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 79450158080 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚766873 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 82209615872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 trainer/global_step‚99 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204858880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7613497505792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7534152867328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚983136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 79361677824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5659030891008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812ºqÜÜ縆O J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚766871 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 79361677824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚766871 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚733048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7613497505792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 79450158080 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚766873 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 82209615872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 trainer/global_step‚99 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204858880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7613497505792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7534152867328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚983136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 79361677824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5659030891008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚803 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _runtime‚ 64.068219956 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚732876 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7526612389850 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 79139229738 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7534135827968 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚506897 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7613603025408 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 82320365056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚983136 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚506901 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7844929074662905e+09zd÷?.¹) _wandbruntime‚64  _runtime‚64xW”&ºŽM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚228197 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5798399939072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 119130624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 80009509348 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7609302533632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7609285494272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5798186698752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 64.470475488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 80147393536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7609302533632 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7601790234212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7609285494272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5715168619008 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777264 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774500 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1002762 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚740329 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213240320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚992956 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚228372 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5715262728704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 79922724338 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774502 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚803 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚511953 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 80235880960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 83137210368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1002762 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7689538414592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚740150 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774500 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 83018079744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0–Θ&»’M LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774500 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 83018079744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚228197 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5798399939072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 119130624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 80009509348 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7609302533632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7609285494272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5798186698752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 64.470475488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 80147393536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7609302533632 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7601790234212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7609285494272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5715168619008 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1777264 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774500 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1002762 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚740329 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213240320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 132997632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚992956 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚228372 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5715262728704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 79922724338 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774502 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚803 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚511953 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 80235880960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 83137210368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1002762 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7689538414592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚740150 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552Åçª.¼) _wandbruntime‚64  _runtime‚64Ù$÷½ñ 0loss/SRqqt2/detection_loss‚7.804363039517125 _step‚7 /loss/FRt1/assignment_loss‚5.323140135189941 1loss/SRqqt1/assignment_loss‚4.228286042480072 1loss/SRqqt1/detection_loss‚1.4449522146865512 /loss/FBt1/detection_loss‚13.621448534168303 /loss/FBt2/detection_loss‚24.893953347435364 epoch‚0 _runtime‚ 64.471007148 .loss/FBt2/assignment_loss‚2.08519014945397 /loss/FRt2/detection_loss‚0.4984249374511583 %loss/total_loss‚72.16483387901647 % _timestamp‚1.7844929078693967e+09 /loss/FRt2/assignment_loss‚4.496887062388338 1loss/SRqqt2/assignment_loss‚4.235099567923435 .loss/FRt1/detection_loss‚1.481242919891653 trainer/global_step‚99 0loss/FBt1/assignment_loss‚2.0518459284305575~&œÆû¾õ 0loss/FBt1/assignment_loss‚2.0518459284305575 0loss/SRqqt2/detection_loss‚7.804363039517125 _step‚7 /loss/FRt1/assignment_loss‚5.323140135189941 1loss/SRqqt1/assignment_loss‚4.228286042480072 1loss/SRqqt1/detection_loss‚1.4449522146865512 /loss/FBt1/detection_loss‚13.621448534168303 /loss/FBt2/detection_loss‚24.893953347435364 epoch‚0 _runtime‚ 64.471007148 .loss/FBt2/assignment_loss‚2.08519014945397 /loss/FRt2/detection_loss‚0.4984249374511583 %loss/total_loss‚72.16483387901647 % _timestamp‚1.7844929078693967e+09 /loss/FRt2/assignment_loss‚4.496887062388338 1loss/SRqqt2/assignment_loss‚4.235099567923435 .loss/FRt1/detection_loss‚1.481242919891653 trainer/global_step‚99ºeg*¿‚(:Ÿ öæôÒ¡ëú¾! disk.dm-0.out‚827.54931640625 disk.dm-1.out‚ 2.0029296875 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.out‚1.515625) disk./.usagePercent‚76.54387836366226 disk.rbd0.in‚0& disk.nvme0c0n1.out‚300.82177734375$ disk.nvme0n1.out‚300.82177734375$ disk.nvme1c1n1.out‚ 532.072265625$ memory_percent‚91.91296100616455) proc.memory.availableMB‚ 1324.98046875" proc.memory.rssMB‚ 2802.5078125 network.sent‚103735$ disk./.usageGB‚1175.139892578125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0) proc.memory.percent‚17.10515022277832 cpu‚25.559491781019055 network.recv‚58872 disk.dm-0.in‚0" disk.nvme1n1.out‚ 532.072265625 disk.md0.in‚0 proc.cpu.threads‚17°)±“À‚(:ˆ öæôÒÛÛª¿ gpu.0.gpu‚81.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚189.552' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚63.184 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H~•°žÁ‚(:“ …çôÒÛÁྠgpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚228.553' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.18433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôW¨£*‚(:Ÿ …çôһ奿" proc.memory.rssMB‚ 2802.5078125 network.sent‚150785 disk.nvme1n1.in‚0 disk.dm-0.out‚1074.380859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ disk.nvme1c1n1.out‚ 570.896484375) proc.memory.percent‚17.10515022277832) disk./.usagePercent‚76.48844579942956 disk.md0.out‚ 2.16796875 disk.dm-1.in‚0% disk.nvme0c0n1.out‚510.3525390625# disk.nvme0n1.out‚510.3525390625 disk.nvme1c1n1.in‚0) proc.memory.availableMB‚ 1324.81640625 cpu‚25.565168807377436" disk.nvme1n1.out‚ 570.896484375 disk.dm-0.in‚0 disk.dm-1.out‚ 2.1513671875 proc.cpu.threads‚17 network.recv‚63779% disk./.usageGB‚1174.2888641357422 disk.md0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚91.91396236419678˜yN.Ã) _wandbruntime‚95  _runtime‚95ì‚t§Äľ ' lr-AdamW/pg1‚5.9767713697039556e-05 ' lr-AdamW/pg2‚5.9767713697039556e-05 trainer/global_step‚149 % _timestamp‚1.7844929386027174e+09 _runtime‚ 95.206485141 _step‚8Çž¨ÿÈÅ _runtime‚ 95.206485141 _step‚8 ' lr-AdamW/pg1‚5.9767713697039556e-05 ' lr-AdamW/pg2‚5.9767713697039556e-05 trainer/global_step‚149 % _timestamp‚1.7844929386027174e+09'„$.Æ) _wandbruntime‚95  _runtime‚95XŸ" Ç¡O GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 118711618048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148321 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1475988 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 118464545444 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11410330761216 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634103 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8588923975168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8465917291008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 118377760546 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E=DeviÍØd£ceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1096060 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1475988 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚336359 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11410330761216 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11291619143168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8588700241920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148323 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 122782950912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11161999429712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11291636182528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11291619143168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129623552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2624309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚9 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11410436280832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚759701 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 95.207158943 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11280481014516 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1095877 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11162016469072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1485780 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11410330761216 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2634103 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8588923975168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8465917291008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 118377760546 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1096060 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1475988 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚336359 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11410330761216 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11291619143168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8588700241920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148323 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 122782950912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11161999429712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11291636182528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11291619143168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129623552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2624309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚9 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11410436280832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚759701 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 95.207158943 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11280481014516 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1095877 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11162016469072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1485780 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1495614 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11366768809472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 119585821184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 123590259712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1485808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2641758 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8522149128704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚149 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1485808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223726080 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11355451210250 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1155952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2651566 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1103267 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 119161255146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1155950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚338510 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11486266143232 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11366768809472 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 119497333760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8645869004800 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1103095 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1155950 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8645645278720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚338342 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚764753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11236289955104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 119497333760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1155952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0CPî¯Ë°M I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1485808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223726080 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11355451210250 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1155952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2651566 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1103267 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 119161255146 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1155950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚338510 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11486266143232 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11366768809472 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 119497333760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8645869004800 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1103095 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1155950 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_ÕtoÇbytes.all.freed‚ 8645645278720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚338342 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚764753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11236289955104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 119497333760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1155952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8522055019008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11366785848832 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11486371670016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _step‚10 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1495614 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1495614 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11366768809472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 119585821184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 123590259712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1485808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2641758 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8522149128704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚149 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YƒŸŽ.Ì) _wandbruntime‚95  _runtime‚95ÞèNüùÍó .loss/FRt2/assignment_loss‚4.52204018425838 2loss/SRqqt2/assignment_loss‚4.1917025653001305 /loss/FRt1/detection_loss‚1.5023354932433297 /loss/FRt2/detection_loss‚0.4047723433300017 1loss/SRqqt1/detection_loss‚1.2841542279134497 0loss/SRqqt2/detection_loss‚6.044843336689138 epoch‚0 trainer/global_step‚149 /loss/FBt1/assignment_loss‚1.202507407434525 /loss/FBt2/assignment_loss‚1.277904470761617 /loss/FBt1/detection_loss‚10.025586080842855 .loss/FBt2/detection_loss‚47.32664758739648 _step‚11 /loss/FRt1/assignment_loss‚5.327661968796709 %loss/total_loss‚87.26434580588273 % _timestamp‚1.7844929390087607e+09 1loss/SRqqt1/assignment_loss‚4.154190139916107 _runtime‚ 95.610478152KÀ¿AýÎ÷ _runtime‚ 95.610478152 .loss/FRt2/assignment_loss‚4.52204018425838 2loss/SRqqt2/assignment_loss‚4.1917025653001305 /loss/FRt1/detection_loss‚1.5023354932433297 /loss/FRt2/detection_loss‚0.4047723433300017 1loss/SRqqt1/detection_loss‚1.2841542279134497 0loss/SRqqt2/detection_loss‚6.044843336689138 epoch‚0 trainer/global_step‚149 /loss/FBt1/assignment_loss‚1.202507407434525 /loss/FBt2/assignment_loss‚1.277904470761617 /loss/FBt1/detection_loss‚10.025586080842855 .loss/FBt2/detection_loss‚47.32664758739648 _step‚11 /loss/FRt1/assignment_loss‚5.327661968796709 %loss/total_loss‚87.26434580588273 % _timestamp‚1.7844929390087607e+09 1loss/SRqqt1/assignment_loss‚4.154190139916107 ¥MÕ ŸÏ‚(:” ”çôÒÊ¡î¾ gpu.0.gpu‚62.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.562333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†7Î+Ђ(:  ”çôÒå««¿ network.recv‚67684) disk./.usagePercent‚76.48848878543359% disk./.usageGB‚1174.2895240783691 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚573.49658203125 disk.dm-0.out‚1085.658203125' proc.memory.availableMB‚ 1324.796875" proc.memory.rssMB‚ 2802.5078125 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚573.49658203125 disk.dm-1.out‚ 2.44970703125 disk.rbd0.in‚0) proc.memory.percent‚17.10515022277832 cpu‚25.569541914515547$ disk.nvme0c0n1.out‚ 519.962890625" disk.nvme0n1.out‚ 519.962890625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚168895 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 2.46484375 disk.dm-0.in‚0$ memory_percent‚91.91408157348633 proc.cpu.threads‚17пùF,Ñ‚(:¡ £çôÒÚã¾% disk.nvme1c1n1.out‚574.1650390625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚574.1650390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚521.06884765625$ disk.nvme0n1.out‚521.06884765625 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 2.85546875 disk.dm-1.out‚ 2.53955078125 network.recv‚72357) disk./.usagePercent‚76.50119922503143$ disk./.usageGB‚1174.484661102295 disk.rbd0.out‚0$ memory_percent‚91.91203117370605( proc.memory.availableMB‚ 1325.1328125" proc.memory.rssMB‚ 2802.5078125) proc.memory.percent‚17.10515022277832 disk.dm-0.in‚0 disk.dm-0.out‚1086.505859375 disk.dm-1.in‚0 cpu‚25.57504652664387 proc.cpu.threads‚17 network.sent‚203128 disk.nvme0n1.in‚0êˆüÊŸÒ‚(:” £çôÒ§‡¢¿ gpu.0.gpu‚70.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚123.821' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.273666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•…ö.0Ó+ _wandbruntime‚126  _runtime‚126Ž¾³˜ÃÔ½ _runtime‚ 126.364488849 _step‚12 & lr-AdamW/pg1‚7.982399346114679e-05 & lr-AdamW/pg2‚7.982399346114679e-05 trainer/global_step‚199 $ _timestamp‚1.784492969761117e+09ݽšçÇÕÁ _runtime‚ 126.364488849 _step‚12 & lr-AdamW/pg1‚7.982399346114679e-05 & lr-AdamW/pg2‚7.982399346114679e-05 trainer/global_step‚199 $ _timestamp‚1.784492969761117e+09 %¥90Ö+ _wandbruntime‚126  _runtime‚126Ìó®'רO TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15207164016640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11272897800704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123332096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15207164016640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15049102458368 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508405 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1968840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15034245814924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 157616291354 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1458094 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15207269536256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚199 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15049119497728 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 163351478272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 158061558272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11272803691008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 158150038528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 158150038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217441792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1457921 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14876542738672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15049102458368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1529771 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 157703076252 % _timestamp‚1.7844929697633264e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1978632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11436372611072 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 163474810368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1978632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1012501 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14876525699312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 158061558272 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1529773 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15034141990666 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360UŒ2+²'جO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15207269536256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚199 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15049119497728 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 163351478272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 158061558272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3498611 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11272803691008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 158150038528 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 158150038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217441792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1457921 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14876542738672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15049102458368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1529771 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 157703076252 % _timestamp‚1.7844929697633264e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1978632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11436372611072 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 163474810368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1978632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1012501 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14876525699312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 158061558272 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1529773 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15034141990666 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15207164016640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11272897800704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123332096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15207164016640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15049102458368 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508405 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1968840 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15034245814924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 157616291354 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1458094 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 õÄ.0Ù+ _wandbruntime‚126  _runtime‚126‚:ÛOV Ú³M E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 _step‚14 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 158847273984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525868 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15124252124672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 164155780608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1465393 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537402 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537402 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15124252124672 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1465211 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚199 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 158486570964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 126.765380445 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚447836 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11493323022848 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 % _timestamp‚1.7844929701634157e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1988466 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15124269164032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15109319835028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537400 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1978660 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 164287494144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1017553 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚447658 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225823232 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15283204925440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3516060 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15283204925440 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 158847273984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11328941419008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15283099398656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1988466 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 158399785954 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760›°¡Ð½&Û·M UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 126.765380445 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚447836 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11493323022848 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 % _timestamp‚1.7844929701634157e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1988466 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15124269164032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15109319835028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537400 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1978660 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 164287494144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1017553 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚447658 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225823232 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15283204925440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3516060 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15283204925440 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 158847273984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11328941419008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15283099398656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1988466 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 158399785954 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 _step‚14 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 158847273984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525868 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15124252124672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 164155780608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1465393 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537402 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537402 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15124252124672 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1465211 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚199 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 158486570964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 ODeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18806602812928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 197499978752 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18591051968912 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2461692 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 197411498496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2471484 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 197411498496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 203914815488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 197499978752 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19003997272064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 196854822162 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 204042341888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18788010615332 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19003997272064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18787906791074 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2471484 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19004102791680 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4382707 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080<ý'.è­O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 196854822162 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 204042341888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚‹/;f…$0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18788010615332 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19003997272064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18787906791074 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2471484 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19004102791680 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4382707 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4372913 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18806602812928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18806602812928 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 197499978752 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18591051968912 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2461692 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 197411498496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2471484 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 197411498496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 203914815488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 197499978752 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19003997272064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360±¿~<0é+ _wandbruntime‚160  _runtime‚160ÝÖi¹&ê³M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 160.049961755 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18881752479232 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚556055 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19080038180864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 204848742400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18881735439872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 197725101772 trainer/global_step‚249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19080038180864 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400170 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2481318 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14340557430784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 197638316762 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 119130624 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 204729611776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18863084635436 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2471512 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1826408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1270357 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1270353 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2481318 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚167 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1826575 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 198197214208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2471512 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18881752479232 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1918852 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 198285701632 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1918850 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536)Aí½&ë·M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 198285701632 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 119130624 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 204729611776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18863084635436 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2471512 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1826408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1270357 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1270353 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2481318 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚167 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1826575 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 198197214208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2471512 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18881752479232 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1918852 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 198285701632 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1918850 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 160.049961755 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18881752479232 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚556055 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19080038180864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 204848742400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18881735439872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 197725101772 trainer/global_step‚249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19080038180864 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400170 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2481318 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14340557430784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 197638316762 BS0ó+ _wandbruntime‚192  _runtime‚192m¼«ÒÆôÀ % _timestamp‚1.7844930361258173e+09 _runtime‚ 192.727658683 _step‚20 ' lr-AdamW/pg1‚0.00011993655298936128 ' lr-AdamW/pg2‚0.00011993655298936128 trainer/global_step‚2991c^ÊõÄ _step‚20 ' lr-AdamW/pg1‚0.00011993655298936128 ' lr-AdamW/pg2‚0.00011993655298936128 trainer/global_step‚299 % _timestamp‚1.7844930361258173e+09 _runtime‚ 192.727658683G|ÒD0ö+ _wandbruntime‚192  _runtime‚192ƒ“G[®'÷¨O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784493036126821e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22800936047104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1518097 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 244474644480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 trainer/global_step‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2180133 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 244604268032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2292673 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2964336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129623552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22305595277872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2292671 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2964336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 236093352970 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 236849918976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚661850 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17131274868736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22564069088768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16886576491008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22564086128128 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22541775415740 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22541671591482 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2954544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2179947 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22800830527488 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2292673 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 236761438720 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22800830527488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 236180137868 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17131051135488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812bªJŸ²'ø¬O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784493036126821e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22800936047104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1518097 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 244474644480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 trainer/global_step‚299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2180133 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 244604268032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2292673 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2964336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129623552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22305595277872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2292671 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2964336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 236093352970 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 236849918976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚661850 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17131274868736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22564069088768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16886576491008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22564086128128 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22541775415740 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22541671591482 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2954544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2179947 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22800830527488 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2292673 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 236761438720 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22800830527488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 236180137868 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17131051135488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360Y­Ü0ù+ _wandbruntime‚193  _runtime‚193üÈN’¹&ú³M WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 _runtime‚ 193.148848137 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5274472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2300302 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223726080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚22 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22639235794432 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22876765909504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚663976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16942808328704 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2974170 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2964364 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2974170 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2187129 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 236876847570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22379868763904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 237635641856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22639235794432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245415149568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22876871436288 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2300300 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2964364 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1523157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245285533184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22379885803264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2187304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1523153 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2300300 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 237635641856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2300302 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 % _timestamp‚1.7844930365467417e+09 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5264664 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22616745611474¢fºÿ?û·M @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22639235794432 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22876765909504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚663976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16942808328704 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2974170 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2964364 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2974170 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2187129 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 236876847570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool'%m~!.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22379868763904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 237635641856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22639235794432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245415149568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22876871436288 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2300300 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2964364 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1523157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245285533184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22379885803264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2187304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1523153 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2300300 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 237635641856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129616384 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2300302 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 % _timestamp‚1.7844930365467417e+09 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5264664 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22616745611474 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 _runtime‚ 193.148848137 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5274472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2300302 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223726080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚22éïË0ü+ _wandbruntime‚193  _runtime‚193zOÜÿýù _step‚23 2loss/SRqqt2/assignment_loss‚2.4357754547020485 %loss/total_loss‚45.16806013554498 1loss/FBt2/assignment_loss‚0.37645990083910874 /loss/FRt2/detection_loss‚0.3691959415641632 1loss/SRqqt1/detection_loss‚0.9970616571457361 .loss/FBt2/detection_loss‚17.96186124239466 _runtime‚ 193.149373488 /loss/FRt2/assignment_loss‚4.543034305134927 2loss/SRqqt1/assignment_loss‚2.2239351619045267 1loss/FBt1/assignment_loss‚0.15806688589078408 0loss/SRqqt2/detection_loss‚5.195583049577244 epoch‚0 /loss/FRt1/assignment_loss‚4.729748435931558 /loss/FRt1/detection_loss‚1.5208881367205784 .loss/FBt1/detection_loss‚4.656449963739639 trainer/global_step‚299 % _timestamp‚1.7844930365477934e+09†Š&þý 1loss/FBt2/assignment_loss‚0.37645990083910874 /loss/FRt2/detection_loss‚0.3691959415641632 1loss/SRqqt1/detection_loss‚0.9970616571457361 .loss/FBt2/detection_loss‚17.96186124239466 _runtime‚ 193.149373488 /loss/FRt2/assignment_loss‚4.543034305134927 2loss/SRqqt1/assignment_loss‚2.2239351619045267 1loss/FBt1/assignment_loss‚0.15806688589078408 0loss/SRqqt2/detection_loss‚5.195583049577244 epoch‚0 /loss/FRt1/assignment_loss‚4.729748435931558 /loss/FRt1/detection_loss‚1.5208881367205784 .loss/FBt1/detection_loss‚4.656449963739639 trainer/global_step‚299 % _timestamp‚1.7844930365477934e+09 _step‚23 2loss/SRqqt2/assignment_loss‚2.4357754547020485 %loss/total_loss‚45.16806013554498ø‡Š-ÿ‚(:¢ îçôÒŽµâ¾% disk.nvme1c1n1.out‚593.9052734375 disk.nvme1n1.in‚0 disk.dm-1.out‚ 3.88134765625# proc.memory.rssMB‚ 2802.51171875 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1323.88671875) proc.memory.percent‚17.10517406463623 proc.cpu.threads‚17 disk.nvme0n1.in‚0# disk.nvme0n1.out‚1694.158203125 disk.md0.in‚0 disk.rbd0.in‚0 network.sent‚314312 disk.nvme0c0n1.in‚0# disk.nvme1n1.out‚593.9052734375 disk.md0.out‚ 4.8203125" disk.dm-0.out‚2273.76708984375 disk.rbd0.out‚0# memory_percent‚91.9196367263794 cpu‚25.615122416144416 network.recv‚92973) disk./.usagePercent‚76.56384424440364$ disk./.usageGB‚1175.446418762207% disk.nvme0c0n1.out‚1694.158203125¯1nf“€‚(:ˆ îçôÒüÖ–¿ gpu.0.gpu‚90.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.038' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚75.346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ióé1‚(:¦ ýçôÒ¬¨ô¾& disk.nvme1c1n1.out‚594.83349609375 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚0$ disk.nvme1n1.out‚594.83349609375 disk.md0.out‚ 5.1484375 disk.nvme0n1.in‚0$ memory_percent‚91.92118644714355( proc.memory.availableMB‚ 1323.6328125# proc.memory.rssMB‚ 2802.51171875) proc.memory.percent‚17.10517406463623 cpu‚25.61184261712173 proc.cpu.threads‚17 network.sent‚360340 network.recv‚97926% disk./.usageGB‚1175.4475021362305% disk.nvme0n1.out‚1695.88427734375 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚2275.62353515625' disk./.usagePercent‚76.563914811023 disk.dm-1.in‚0 disk.dm-1.out‚ 3.97119140625 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1695.88427734375e¸àÙ“‚‚(:ˆ ýçôÒ¤ö¹¿ gpu.0.gpu‚98.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.458' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì]n2ƒ‚(:§ ŒèôÒÔÉý¾ disk.md0.out‚5.484375 disk.dm-1.out‚ 4.10791015625 disk.rbd0.in‚0 disk.nvme0n1.in‚0# proc.memory.rssMB‚ 2802.51171875) proc.memory.percent‚17.10517406463623 cpu‚25.60689668519209 proc.cpu.threads‚17$ disk./.usageGB‚1175.500804901123 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚595.49462890625 disk.md0.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚2288.55322265625 disk.rbd0.out‚0) proc.memory.availableMB‚ 1324.38671875 network.recv‚101968) disk./.usagePercent‚76.56738673838939 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚595.49462890625 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚1709.03369140625# memory_percent‚91.9165849685669 network.sent‚367437 disk.dm-1.in‚0% disk.nvme0n1.out‚1709.03369140625ßG£qŸ„‚(:” ŒèôÒäÔ¬¿ gpu.0.gpu‚80.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚158.867' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚52.955666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â³!0…+ _wandbruntime‚226  _runtime‚226§…j™Ä†¾ trainer/global_step‚349 % _timestamp‚1.7844930694348133e+09 _runtime‚ 226.037207556 _step‚24 & lr-AdamW/pg1‚0.0001399928327534685 & lr-AdamW/pg2‚0.0001399928327534685APïȇ _runtime‚ 226.037207556 _step‚24 & lr-AdamW/pg1‚0.0001399928327534685 & lr-AdamW/pg2‚0.0001399928327534685 trainer/global_step‚349 % _timestamp‚1.7844930694348133e+09øeuË0ˆ+ _wandbruntime‚226  _runtime‚226qƒw®'‰¨O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26321552403968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26020121547472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26597663782912 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2540113 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221636096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚769216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 $ _timestamp‚1.784493069435532e+09 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 284997233664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 276111378944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19693557000704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 285124760064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26321569443328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 276199859200 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6131311 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3457188 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19693462891008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26597769302528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1770901 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127526400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26020104508112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2674123 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1770897 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 275331883778 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚769394 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚25 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 276199859200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3457188 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26321569443328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 276111378944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2674121 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _runtime‚ 226.037402336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19978681760768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19978460124672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26597663782912 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682¢òð¾ñŠ¬O PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2540295 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26295540216148 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 275418668676 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26295436391890 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26597769302528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2674123 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2674121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26020104508112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2674123 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1770897 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 275331883778 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚769394 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚25 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 276199859200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3457188 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26321569443328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 276111378944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2674121 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _runtime‚ 226.037402336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19978681760768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19978460124672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26597663782912 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26321552403968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26020121547472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26597663782912 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2540113 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221636096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚769216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 $ _timestamp‚1.784493069435532e+09 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 284997233664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 276111378944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19693557000704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 285124760064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26321569443328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatMlÞ ÁsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 276199859200 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6131311 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3457188 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19693462891008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26597769302528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86784898 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1770901 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127526400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26396719109632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269121024 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3457216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 276115378378 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2681752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2681752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26396702070272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2681750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20035406038016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3467022 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚771347 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 276202163388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26370614236252 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6148774 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19749600619008 $ _timestamp‚1.784493069850858e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26673599164928 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3457216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230017536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26673704691712 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 _step‚26 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3467022 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚771530 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26094412072864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26396719109632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26396702070272 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2547300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26370510411882 _runtime‚ 226.452869163 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 276985582080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 285941326848 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6138966 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1775953 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2681752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26396702070272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2681750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88487424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20035406038016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3467022 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚771347 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 276202163388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26370614236252 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6148774 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19749600619008 $ _timestamp‚1.784493069850858e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26673599164928 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3457216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230017536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26673704691712 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 _step‚26 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301697536 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3467022 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚771530 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26094412072864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26396719109632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26396702070272 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2547300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26370510411882 _runtime‚ 226.452869163 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 276985582080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370606080 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 285941326848 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6138966 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1775953 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0çXg0Ž+ _wandbruntime‚226  _runtime‚226… ƒ#û 1loss/FBt2/assignment_loss‚0.21265985590030645 /loss/FBt2/detection_loss‚17.539222985792478 epoch‚0 1loss/SRqqt2/assignment_loss‚2.081712137739002 1loss/FBt1/assignment_loss‚0.24249324562181282 /loss/FRt2/detection_loss‚0.3904460329778673 1loss/SRqqt1/detection_loss‚1.1057028165110028 _step‚27 .loss/FBt1/detection_loss‚4.594159840962907 trainer/global_step‚349 % _timestamp‚1.7844930698518648e+09 %loss/total_loss‚41.76209114190796 0loss/FRt1/assignment_loss‚3.9630080634897404 0loss/FRt2/assignment_loss‚4.5071150387114605 /loss/FRt1/detection_loss‚1.3967602590268309 0loss/SRqqt2/detection_loss‚4.058250958395258 _runtime‚ 226.453390414 2loss/SRqqt1/assignment_loss‚1.6705599067792947œÜ_ÿ /loss/FBt2/detection_loss‚17.539222985792478 epoch‚0 1loss/SRqqt2/assignment_loss‚2.081712137739002 1loss/FBt1/assignment_loss‚0.24249324562181282 /loss/FRt2/detection_loss‚0.3904460329778673 1loss/SRqqt1/detection_loss‚1.1057028165110028 _step‚27 .loss/FBt1/detection_loss‚4.594159840962907 trainer/global_step‚349 % _timestamp‚1.7844930698518648e+09 %loss/total_loss‚41.76209114190796 0loss/FRt1/assignment_loss‚3.9630080634897404 0loss/FRt2/assignment_loss‚4.5071150387114605 /loss/FRt1/detection_loss‚1.3967602590268309 0loss/SRqqt2/detection_loss‚4.058250958395258 _runtime‚ 226.453390414 2loss/SRqqt1/assignment_loss‚1.6705599067792947 1loss/FBt2/assignment_loss‚0.21265985590030645ÙZ7Ç1‘‚(:¦ ›èôÒá´‚¿& disk.nvme0c0n1.out‚1711.1103515625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2802.51171875 cpu‚25.604511711872213 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# memory_percent‚91.9175386428833 proc.cpu.threads‚17' disk./.usagePercent‚76.485836325358! disk.dm-0.out‚2290.5322265625 disk.dm-1.out‚ 4.24462890625 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚1711.1103515625& disk.nvme1c1n1.out‚596.33056640625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚596.33056640625 network.sent‚383967 network.recv‚106482 disk.rbd0.in‚0 disk.nvme0n1.in‚0) proc.memory.availableMB‚ 1324.23046875) proc.memory.percent‚17.10517406463623% disk./.usageGB‚1174.2488021850586 disk.md0.out‚ 5.85546875 disk.dm-0.in‚0>FG$’‚(:’ ›èôÒ gpu.0.gpu‚39.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚213.283' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.09433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äbåN3“‚(:¨ ªèôÒúáä¾ network.sent‚420109 disk.dm-1.out‚ 4.35986328125& disk.nvme0c0n1.out‚1712.0419921875# proc.memory.rssMB‚ 2802.51171875 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚596.88525390625 disk.nvme1n1.in‚0 disk.md0.out‚ 6.17578125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚1712.0419921875$ disk.nvme1n1.out‚596.88525390625 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 1324.1953125 cpu‚25.599658142227277! disk.dm-0.out‚2291.2119140625 network.recv‚111232) disk./.usagePercent‚76.48585371853882% disk./.usageGB‚1174.2490692138672 disk.dm-0.in‚0$ memory_percent‚91.91775321960449) proc.memory.percent‚17.10517406463623ÙåÔ»ž”‚(:“ ªèôÒ·±”¿ gpu.0.gpu‚62.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.842' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.28066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü³}0•+ _wandbruntime‚259  _runtime‚259çÎl)Å–¿ ' lr-AdamW/pg1‚0.00016004911251757574 ' lr-AdamW/pg2‚0.00016004911251757574 trainer/global_step‚399 $ _timestamp‚1.784493102715678e+09 _runtime‚ 259.317510452 _step‚28z‰$É—Ã _step‚28 ' lr-AdamW/pg1‚0.00016004911251757574 ' lr-AdamW/pg2‚0.00016004911251757574 trainer/global_step‚399 $ _timestamp‚1.784493102715678e+09 _runtime‚ 259.317510452æwu—0˜+ _wandbruntime‚259  _runtime‚259ó2ñh™©O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88480256 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2899999 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 _step‚29 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚876298 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 314657199484 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 315549799424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3950040 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30394497038336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30049201192298 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 325648549888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6995819 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30394602557952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2023697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3940248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30079052758528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30394497038336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30079035719168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3055573 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3940248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 314575ˆG0414586 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30394602557952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 315461319168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 _runtime‚ 259.318557634 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2023701 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 325523120640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 315549799424 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚876123 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30049305016556 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚399 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7005613 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30079052758528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2899999 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103824258 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 _step‚29 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚876298 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 314657199484 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 315549799424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3950040 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30394497038336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30049201192298 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267415751 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 325648549888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6995819 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30394602557952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2023697 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3940248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301697536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30079052758528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30394497038336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106882647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301697536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30079035719168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105519616 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3055573 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3940248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370606080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 137190912 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 314570414586 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30394602557952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 315461319168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 _runtime‚ 259.318557634 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269121024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2023701 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 325523120640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 315549799424 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚876123 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30049305016556 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚399 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269121024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105519616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88480256 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7005613 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30079052758528 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 259.738886296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30154185385472 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30154185385472 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30124275212290 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3063202 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3950068 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3950068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30124379036660 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30154202424832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22556581128704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135907840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30154202424832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 # _timestamp‚1.78449310313682e+09 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚878289 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22556487019008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29808921303104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 316247034880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 316335522304 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22883041895424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 315440694196 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 316335522304 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3063200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3063200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2907042 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7013268 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚878467 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3063202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7023076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2907224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230017536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22882811877888 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 326460766720À?Ú}ü!µM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30154185385472 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30124275212290 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3063202 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3950068 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3950068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30124379036660 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106882647 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30154202424832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88487424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269121024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22556581128704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135907840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105526784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30154202424832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 # _timestamp‚1.78449310313682e+09 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚878289 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22556487019008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103824370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29808921303104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 316247034880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 316335522304 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22883041895424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 137190912 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 315440694196 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301697536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 316335522304 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3063200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3063200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2907042 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7013268 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚878467 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86785010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3063202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7023076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2907224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230017536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22882811877888 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 326460766720 REDeviceStatsMonitor.on_train_batch_end/fŠ¿active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105526784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267415751 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 259.738886296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30154185385472 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806ùJ;0ž+ _wandbruntime‚259  _runtime‚259ú YâŸû /loss/FRt2/assignment_loss‚4.057636713805022 2loss/SRqqt2/assignment_loss‚1.8647100745113032 1loss/FBt2/assignment_loss‚0.14583050924044133 .loss/FBt1/detection_loss‚4.624993145255244 /loss/FBt2/detection_loss‚16.933772934081016 0loss/FRt1/assignment_loss‚3.4120070535864606 1loss/SRqqt2/detection_loss‚3.4644903424329416 %loss/total_loss‚38.99009131961125 epoch‚0 _runtime‚ 259.739393577 _step‚31 2loss/SRqqt1/assignment_loss‚1.5466166840425666 1loss/FBt1/assignment_loss‚0.16290577165050993 trainer/global_step‚399 % _timestamp‚1.7844931031378791e+09 .loss/FRt1/detection_loss‚1.439638883342979 /loss/FRt2/detection_loss‚0.3645220025204967 1loss/SRqqt1/detection_loss‚0.9729672051422718(£¨  ÿ 1loss/SRqqt1/detection_loss‚0.9729672051422718 /loss/FRt2/assignment_loss‚4.057636713805022 2loss/SRqqt2/assignment_loss‚1.8647100745113032 1loss/FBt2/assignment_loss‚0.14583050924044133 .loss/FBt1/detection_loss‚4.624993145255244 /loss/FBt2/detection_loss‚16.933772934081016 0loss/FRt1/assignment_loss‚3.4120070535864606 1loss/SRqqt2/detection_loss‚3.4644903424329416 %loss/total_loss‚38.99009131961125 epoch‚0 _runtime‚ 259.739393577 _step‚31 2loss/SRqqt1/assignment_loss‚1.5466166840425666 1loss/FBt1/assignment_loss‚0.16290577165050993 trainer/global_step‚399 % _timestamp‚1.7844931031378791e+09 .loss/FRt1/detection_loss‚1.439638883342979 /loss/FRt2/detection_loss‚0.36452200252049675î‚2¡‚(:§ ¹èôҵ᱿ network.sent‚436270 disk.md0.in‚0 disk.dm-0.out‚ 2298.30078125 disk.dm-1.out‚ 4.47314453125# proc.memory.rssMB‚ 2882.05859375* proc.memory.percent‚17.590689659118652 cpu‚25.70259096260584 network.recv‚115307% disk./.usageGB‚1174.2490005493164& disk.nvme0c0n1.out‚1719.1865234375$ disk.nvme0n1.out‚1719.1865234375 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚597.60888671875 disk.md0.out‚ 6.48046875 disk.dm-1.in‚0) disk./.usagePercent‚76.48584924600661 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚597.60888671875) proc.memory.availableMB‚ 1243.89453125 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0$ memory_percent‚92.40787029266357 proc.cpu.threads‚17>–<¢‚(:’ ¹èôÒ–ùÞ¿ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.263' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.08766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ÅØÎH£bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPn {H¤bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP*JÝ£H¥bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP™X46@¦b; particle/accuracy_1_of_5"trainer/global_step2:JPk×€N§bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP+²¢N¨bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÐoJ©bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‘eæÐJªbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPO`‰lJ«bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPD`·~N¬bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPK¿”MJ­bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP®ßÉH®bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPsW N¯bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¡_¾;°b6 jet/accuracy_1_of_3"trainer/global_step2:JP-#@±b; particle/accuracy_3_of_3"trainer/global_step2:JP“ñ<9²b4 particle/accuracy"trainer/global_step2:JP•0£H³bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP[;qêJ´bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPñüGü;µb6 jet/accuracy_4_of_4"trainer/global_step2:JPÅ&H¶bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPþc8ÅJ·bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPêtÞïH¸bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPeV¤íH¹bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPxv0^HºbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPä›_Ç@»b; particle/accuracy_1_of_3"trainer/global_step2:JPíSÄEJ¼bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPY#òöN½bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP^f.ÂH¾bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP…¿H¿bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPzÏ|¤HÀbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPVÛNÁbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPFvnã;Âb6 jet/accuracy_1_of_4"trainer/global_step2:JPÇ̸îJÃbE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP´'!JÄbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP3å NÅbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPo ³çNÆbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPOÉuÒJÇbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP$_fãHÈbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPð÷°¼NÉbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP+QGü;Êb6 jet/accuracy_1_of_5"trainer/global_step2:JP÷ HËbC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP\âlJÌbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPEßKÐNÍbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP·M¤;Îb6 loss/val_total_loss"trainer/global_step2:JPÕÄ ËNÏbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPøCHîHÐbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPD-HÑbC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPŒ?j¥HÒbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP˜‰d ;Ób6 jet/accuracy_2_of_4"trainer/global_step2:JPž÷„ANÔbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPi’ÊHÕbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPf«öJÖbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPˆLªÄ@×b; particle/accuracy_1_of_1"trainer/global_step2:JPi¼TÞ@Øb; particle/accuracy_2_of_3"trainer/global_step2:JPK-TrNÙbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPé…ØÆHÚbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP#ý0ØNÛbI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP™»ŒiJÜbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP åƒàJÝbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‘¯×“JÞbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPHàŽyJßbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPâ÷¥[NàbI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPI¿]NábI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÖfÍWJâbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP Ú2«HãbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPàåÒJäbE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPoò ÖNåbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPƒO{øNæbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP =¡¶NçbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPHÒi¼HèbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPyORºHébC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP×HÑçJêbE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPžSÒJëbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPd[‚Ù@ìb; particle/accuracy_4_of_4"trainer/global_step2:JP£\âJíbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¢’åëJîbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPû¾ HïbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¬-~ HðbC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP’Ö˜HñbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPk'7àHòbC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP†Ø®öNóbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPZïmkNôbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP‚½MJõbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÄïJöbE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPøH÷bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP”Z JøbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¸sn?@ùb; particle/accuracy_4_of_5"trainer/global_step2:JP,éõ-JúbE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP|Ï‚æJûbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPOöYJübE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP–SJýbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPï‡BÑNþbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÎSÿHÿbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP¼µxæJ€bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPjrŒÍNbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP©NchH‚bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPñ GJƒbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPöw¾N„bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPu¨LN…bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPUoJ†bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP± UkH‡bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP`Y?HˆbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP‡ qþJ‰bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÕÊ3æNŠbI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPk?Ì>J‹bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP\··"@Œb; particle/accuracy_1_of_2"trainer/global_step2:JPÕ]*ÖJbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP Œ\RHŽbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP·ZÂÒHbC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPwô´4NbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPŸ¥EJ‘bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPGLçH’bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP]«Ä¦N“bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÁwŽH”bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¹­J•bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP4‰¥çH–bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¥n…J—bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPý¶›N˜bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP‡é§H™bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÿª‚cJšbE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPù.º;›b6 jet/accuracy_3_of_5"trainer/global_step2:JP<– 7@œb; particle/accuracy_5_of_5"trainer/global_step2:JPÅ!‘]NbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP hT±NžbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP…RH‹HŸbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPEÞÌJ bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPß‚jJ¡bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP\]j…;¢b6 jet/accuracy_5_of_5"trainer/global_step2:JP(ùßaH£bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPmN:IN¤bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP;„é?J¥bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP/³;@¦b; particle/accuracy_3_of_4"trainer/global_step2:JP¶{›ëH§bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¡_cJ¨bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPNt9éH©bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPKõ|'HªbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPLÍ©jJ«bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPèùXJ¬bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¶òµN­bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPí£–ZJ®bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPr®\;¯b6 jet/accuracy_3_of_4"trainer/global_step2:JP5UXIH°bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPˆó»‰H±bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¡uN²bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPGî<ºJ³bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPý˜J´bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP†“>PJµbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPËÊÿ>J¶bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPµ!pN·bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPGÂQJ¸bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPó³ç°J¹bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP=‡ðQJºbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP~ZßJ»bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPvÿô¶H¼bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¾HÂWJ½bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPqŸ¥H¾bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPW`ÐH¿bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPôclNÀbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPë$úûNÁbI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÒñ†JÂbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPƒÓsSHÃbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÜ™o{NÄbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPšJÅbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP9M‹HÆbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÕ„>;Çb6 jet/accuracy_2_of_3"trainer/global_step2:JPÅ€M@Èb; particle/accuracy_1_of_4"trainer/global_step2:JPqè²¾JÉbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPTLHÊbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPTêâàJËbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPlùF]JÌbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‰Û¡NÍbI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPoÔkJÎbE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPk rö@Ïb; particle/accuracy_2_of_4"trainer/global_step2:JPÛîÔø8Ðb3 particle/f_score"trainer/global_step2:JPCÀçÒHÑbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPãæÞ»HÒbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP2Ò…ÊJÓbE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP„ ž‘<Ôb7 particle/specificity"trainer/global_step2:JP"²„èJÕbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPPLÄôJÖbE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP~ÒiˆN×bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÅ d°HØbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP•NtJÙbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP®xZHÚbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPg§JÛbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP p¬HÜbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP)Ä—æJÝbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP. §Ï;Þb6 jet/accuracy_1_of_2"trainer/global_step2:JPÙ]Y:GßbB validation_average_jet_accuracy"trainer/global_step2:JPkøJàbE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPËO2ÅJábE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP;g.HâbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPMTNãbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP™¢gJäbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPCéJåbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPL ¸êHæbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPòÕcw;çb6 jet/accuracy_4_of_5"trainer/global_step2:JPâ÷.NèbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP$ªW¾NébI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPëµC@êb; particle/accuracy_3_of_5"trainer/global_step2:JPA‡¶NëbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPr‡ù_JìbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPóUæbJíbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP…ôúMNîbI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP+a$2JïbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPOÏÓHðbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÙ÷ÉÝJñbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPo™Ø4NòbI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÝfÍ;ób6 jet/accuracy_1_of_1"trainer/global_step2:JPzÖ@ôb; particle/accuracy_2_of_5"trainer/global_step2:JPn¡µÉHõbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP˜LwNöbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPýà@J÷bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP2«2QJøbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP[´âJùbE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÆ$0JúbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP2!•FJûbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPxFÖðJübE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPŽoWÒ;ýb6 jet/accuracy_2_of_5"trainer/global_step2:JP’r0JþbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPåÖå JÿbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÊž$AH€bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPUŠø7JbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPx'´*N‚bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP ôؾHƒbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP|°sDJ„bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPdÊH…bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPôêjøH†bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP ín0J‡bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPê @-HˆbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP9è¾H‰bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPãÚNŠbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¹ýâN‹bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP‹võöNŒbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¯èTÒJbE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP>›bmJŽbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPc^tAHbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPŸ7·JbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP‚y4QN‘bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPácq£;’b6 jet/accuracy_3_of_3"trainer/global_step2:JPŸDïH“bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP›9"¦H”bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÒrH•bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÜBáH–bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP›97H—bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPg)UÑN˜bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPØC]Ú<™b7 particle/sensitivity"trainer/global_step2:JPve?PHšbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPËÃÜH›bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPí*/JœbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP–³¿œNbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÊÖÿJžbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÒº‹JŸbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÞ ÔJ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÈ;¡b6 jet/accuracy_2_of_2"trainer/global_step2:JPÆ;N¢bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPRÄJ£bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPúkü¹@¤b; particle/accuracy_2_of_2"trainer/global_step2:JPuè«J¥bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP‰ø|H¦bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPC¾TqH§bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPgÆMyH¨bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPUyè»J©bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPerˆ\HªbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPss¬H«bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPË‹©\H¬bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPa_DJ­bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP=‡Ñ†H®bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPBgúJ¯bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP†¡ÉN°bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPwíH±bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPìÉ|N²bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP«ÖŽN³bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP.¼HH´bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¨ôÖJµbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÛ›Ù¸J¶bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPæ®`H·bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPá‘®KJ¸bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP†±]™N¹bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÿ„oJºbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPþhIN»bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPt. H¼bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÜÓ­ÒN½bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPæÀ‘H¾bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPE õN¿bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPæ@HÀbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÊ<ÜNÁbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP­6ª NÂbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP퉛Ê0Ã+ _wandbruntime‚273  _runtime‚273TUì9(Äây 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.4167437255382538 7validation_average_jet_accuracy‚0.15755781531333923 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.02864583395421505 epoch‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.16062049567699432 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.21059516072273254 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0028539206832647324 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7065920233726501 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2372579276561737 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_1_of_1‚0.12850210070610046 particle/accuracy_2_of_5‚1 _step‚32 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9947288036346436 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7645215392112732 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179729223251343 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.16217586398124695 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.29891499876976013 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20490840077400208 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.15849219262599945 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9968022704124451 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16209805011749268 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12078404426574707 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884464740753174 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9980952143669128 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03440292552113533 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.14918746054172516 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.20906352996826172 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.21994030475616455 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.10174339264631271 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,jet/accuracy_3_of_3‚0.012655583210289478 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07975506782531738 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894269108772278 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.12395721673965454 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6003094911575317 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08115352690219879 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972149133682251 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3301970958709717 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.16252079606056213 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 ,"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.375 +jet/accuracy_2_of_2‚0.05010303482413292 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11648941040039062 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945375323295593 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16716279089450836 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16180312633514404 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2152777910232544 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9980952143669128 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0009523809421807528 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5631369352340698 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16041916608810425 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18470928072929382 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0020890217274427414 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.048960547894239426 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09026278555393219 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10309666395187378 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04669305682182312 8 PurityÈ{±À&/2FRt1SRqqt0FBt/FRt_purity‚0.09494269639253616 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10309666395187378 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.21516193449497223 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03615642338991165 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.28646227717399597 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963312745094299 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 )jet/accuracy_1_of_3‚0.536312460899353 -particle/accuracy_3_of_3‚0.99858158826828 (particle/accuracy‚0.7809089422225952 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005087094847112894 jet/accuracy_4_of_4‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875453114509583 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0044419546611607075 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.12087690085172653 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0416666679084301 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.20576103031635284 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9987789988517761 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08149729669094086 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.5156030654907227 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.22361521422863007 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08396390080451965 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963312745094299 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08328939229249954 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_1_of_5‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.20696189999580383 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *loss/val_total_loss‚30.934402072785115 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1476207971572876 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.13135893642902374 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958171844482422 +jet/accuracy_2_of_4‚0.11894755065441132 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1471066027879715 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.04849814251065254 particle/accuracy_1_of_1‚1 /particle/accuracy_2_of_3‚0.9994933605194092 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05562373995780945 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.19509780406951904 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 trainer/global_step‚399 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11290232092142105 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.13408920168876648 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 $ _timestamp‚1.784493116920102e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.12627050280570984 /particle/accuracy_4_of_4‚0.9899720549583435 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ,"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.20190772414207458 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04669305682182312 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06432171911001205 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.12627050280570984 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.11822759360074997 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0021174270659685135 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0005537098622880876 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7554159760475159 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06579342484474182 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.18263418972492218 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1212640255689621 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1359599381685257 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3727447986602783 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.042500000447034836 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9750903844833374 jet/accuracy_3_of_5‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17189332842826843 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0009718172950670123 jet/accuracy_5_of_5‚0 & Purity/*FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.11172810196876526 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.21059516072273254 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17247043550014496 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.23131218552589417 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7713978886604309 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922817945480347 ,jet/accuracy_3_of_4‚0.011249065399169922 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976859092712402 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.047639574855566025 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2183544933795929 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1640625 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.21462103724479675 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12734316289424896 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3801981210708618 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004516381770372391 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.20948322117328644 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07380522042512894 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9901837110519409 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04471202939748764 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04556696116924286 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.14646753668785095 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17040899395942688 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005350454594008625 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03145641088485718 +jet/accuracy_2_of_3‚0.12769348919391632 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3168870210647583 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44999998807907104 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15769197046756744 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.30183640122413635 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.16001330316066742 particle/accuracy_2_of_4‚1 'particle/f_score‚0.5406110882759094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16898857057094574 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.03147434815764427 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8302909135818481 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.18943756818771362 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 _runtime‚ 273.52248175 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.16617538034915924 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13196277618408203 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9934853315353394 ("Purity/1FRt2SRqqt1FBt/event_purity‚0콶Öì<Åæy 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11648941040039062 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945375323295593 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16716279089450836 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16180312633514404 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2152777910232544 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9980952143669128 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0009523809421807528 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5631369352340698 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16041916608810425 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18470928072929382 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0020890217274427414 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.048960547894239426 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09026278555393219 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10309666395187378 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04669305682182312 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09494269639253616 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10309666395187378 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.21516193449497223 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03615642338991165 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.28646227717399597 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963312745094299 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 )jet/accuracy_1_of_3‚0.536312460899353 -particle/accuracy_3_of_3‚0.99858158826828 (particle/accuracy‚0.7809089422225952 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005087094847112894 jet/accuracy_4_of_4‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875453114509583 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0044419546611607075 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.12087690085172653 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0416666679084301 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.20576103031635284 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9987789988517761 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08149729669094086 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.5156030654907227 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.22361521422863007 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08396390080451965 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963312745094299 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08328939229249954 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_1_of_5‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.20696189999580383 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *loss/val_total_loss‚30.934402072785115 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1476207971572876 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.13135893642902374 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958171844482422 +jet/accuracy_2_of_4‚0.11894755065441132 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1471066027879715 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.04849814251065254 particle/accuracy_1_of_1‚1 /particle/accuracy_2_of_3‚0.9994933605194092 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05562373995780945 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.19509780406951904 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 trainer/global_step‚399 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.11290232092142105 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.13408920168876648 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 $ _timestamp‚1.784493116920102e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.12627050280570984 /particle/accuracy_4_of_4‚0.9899720549583435 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ,"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.20190772414207458 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04669305682182312 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06432171911001205 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.12627050280570984 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.11822759360074997 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0021174270659685135 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0005537098622880876 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7554159760475159 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06579342484474182 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.18263418972492218 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1212640255689621 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1359599381685257 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3727447986602783 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.042500000447034836 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9750903844833374 jet/accuracy_3_of_5‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17189332842826843 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0009718172950670123 jet/accuracy_5_of_5‚0 & Purity/*FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.11172810196876526 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.21059516072273254 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17247043550014496 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.23131218552589417 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7713978886604309 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922817945480347 ,jet/accuracy_3_of_4‚0.011249065399169922 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976859092712402 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.047639574855566025 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2183544933795929 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1640625 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.21462103724479675 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12734316289424896 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3801981210708618 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004516381770372391 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.20948322117328644 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07380522042512894 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9901837110519409 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04471202939748764 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04556696116924286 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.14646753668785095 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17040899395942688 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005350454594008625 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03145641088485718 +jet/accuracy_2_of_3‚0.12769348919391632 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3168870210647583 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44999998807907104 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15769197046756744 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.30183640122413635 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.16001330316066742 particle/accuracy_2_of_4‚1 'particle/f_score‚0.5406110882759094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16898857057094574 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.03147434815764427 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8302909135818481 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.18943756818771362 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 _runtime‚ 273.52248175 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.16617538034915924 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13196277618408203 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9934853315353394 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.4167437255382538 7validation_average_jet_accuracy‚0.15755781531333923 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.02864583395421505 epoch‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.16062049567699432 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.21059516072273254 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0028539206832647324 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7065920233726501 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2372579276561737 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_1_of_1‚0.12850210070610046 particle/accuracy_2_of_5‚1 _step‚32 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9947288036346436 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7645215392112732 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179729223251343 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.16217586398124695 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.29891499876976013 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20490840077400208 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.15849219262599945 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9968022704124451 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16209805011749268 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12078404426574707 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884464740753174 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9980952143669128 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03440292552113533 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.14918746054172516 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.20906352996826172 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.21994030475616455 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.10174339264631271 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,jet/accuracy_3_of_3‚0.012655583210289478 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07975506782531738 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894269108772278 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.12395721673965454 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6003094911575317 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08115352690219879 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972149133682251 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3301970958709717 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.16252079606056213 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 ,"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.375 +jet/accuracy_2_of_2‚0.05010303482413292 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 &úÄó%Æ‚(:š ÈèôÒп# memory_percent‚93.2241439819336 network.recv‚119997% disk./.usageGB‚1174.2490196228027 disk.md0.in‚0 disk.nvme0c0n1.in‚0" disk.nvme0n1.out‚ 1720.17578125# proc.memory.rssMB‚ 2887.63671875 cpu‚25.732620339025573 network.sent‚470723) disk./.usagePercent‚76.48585048837666 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 1720.17578125 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 725.994140625 disk.md0.out‚6.921875 disk.dm-0.out‚ 2298.89453125 disk.dm-1.out‚ 132.298828125 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 1110.15625* proc.memory.percent‚17.624735832214355 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme1n1.out‚ 725.9941406251°œÇ‚(:‘ ÈèôÒ„í¿ gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚220.444' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.48133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®8w,È‚(:¡ ×èôÒÄ£¿ cpu‚25.723115690563972 proc.cpu.threads‚17 network.recv‚124683 disk.rbd0.out‚0 disk.nvme0n1.in‚0" disk.nvme1n1.out‚ 726.892578125$ memory_percent‚93.22624206542969% proc.memory.availableMB‚ 1109.8125% disk./.usageGB‚1174.2490844726562 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 726.892578125 disk.dm-0.in‚0" disk.dm-0.out‚2304.26513671875 disk.dm-1.out‚ 132.337890625* proc.memory.percent‚17.624759674072266 network.sent‚528536 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚1725.04638671875 disk.md0.in‚0 disk.md0.out‚7.078125 disk.dm-1.in‚0! proc.memory.rssMB‚ 2887.640625) disk./.usagePercent‚76.48585471243486% disk.nvme0n1.out‚1725.04638671875 disk.nvme1n1.in‚0 ÞAÉ‚(:„ ×èôÒú§Ð¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.51' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.17 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ÅЀ0Ê+ _wandbruntime‚307  _runtime‚307¡âCíÆËÀ % _timestamp‚1.7844931509149256e+09 _runtime‚ 307.516779097 _step‚33 ' lr-AdamW/pg1‚0.00016044776378123916 ' lr-AdamW/pg2‚0.00016044776378123916 trainer/global_step‚449Í’-ÂÊÌÄ ' lr-AdamW/pg1‚0.00016044776378123916 ' lr-AdamW/pg2‚0.00016044776378123916 trainer/global_step‚449 % _timestamp‚1.7844931509149256e+09 _runtime‚ 307.516779097 _step‚33!^óÎE0Í+ _wandbruntime‚307  _runtime‚307{Rº~›Î¯O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35203246816256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4525454 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558821 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369224202795 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558823 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2384752 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34791952112077 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2384756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34422641122864 ]JDevAu7îiceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26492541839360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26110600896000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26492337159168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚ 307.517931489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558823 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35203246816256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7844931509160397e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 _step‚34 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 370240485888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34833006330368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4515313 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34833023369728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 381736263168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34791848286299 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34422624083504 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 370329144832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3414052 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3413863 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26110695005696 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 381846833664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8074134 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35203352514560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682$½›¹'ϳO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2384756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34422641122864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26492541839360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26110600896000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26492337159168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚ 307.517931489 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558823 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35203246816256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7844931509160397e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 _step‚34 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 370240485888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34833006330368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4515313 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34833023369728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 381736263168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34791848286299 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34422624083504 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 370329144832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3414052 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3413863 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26110695005696 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 381846833664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8074134 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35203352514560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35203246816256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4525454 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558821 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369224202795 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558823 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2384752 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34791952112077 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0"Q--Ž0Ð+ _wandbruntime‚307  _runtime‚307,H^<¾&ѸM O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4535288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34496914608896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34908155996672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 382663367680 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2389812 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4535288 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3566450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8101740 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34908173036032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1031407 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26166738624000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34908155996672 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 382546512896 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1031233 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3421041 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34867026132181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26549496101376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26166832733696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116854784 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8091583 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35279287903744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4525133 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚35 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚ 307.93210989 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4525133 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2389812 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4535288 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3566450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8101740 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34908173036032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1031407 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26166738624000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34908155996672 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 382546512896 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1031233 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3421041 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34867026132181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26549496101376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26166832733696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116854784 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8091583 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JH´£¦ BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35279287903744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4525133 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚35 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚ 307.93210989 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4525133 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4535288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34496914608896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34908155996672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 382663367680 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960#þ0¦50Ó+ _wandbruntime‚307  _runtime‚307â CÔû 0loss/FRt1/assignment_loss‚3.1204173388304532 %loss/total_loss‚41.72815038278236 /loss/FRt2/assignment_loss‚4.028909098408559 2loss/SRqqt1/assignment_loss‚1.4616626745314925 /loss/FRt1/detection_loss‚1.3659801868347343 /loss/FRt2/detection_loss‚0.3916557545176613 .loss/FBt1/detection_loss‚5.118257082556268 $ _timestamp‚1.784493151331872e+09 1loss/FBt1/assignment_loss‚0.18214910073729698 1loss/SRqqt1/detection_loss‚0.9898733363168734 1loss/SRqqt2/detection_loss‚3.1388892171962652 epoch‚1 trainer/global_step‚449 _runtime‚ 307.933478772 _step‚36 2loss/SRqqt2/assignment_loss‚1.7630507499433872 1loss/FBt2/assignment_loss‚0.11041165119446682 /loss/FBt2/detection_loss‚20.056894191714896ë3îÕÿ 1loss/FBt1/assignment_loss‚0.18214910073729698 1loss/SRqqt1/detection_loss‚0.9898733363168734 1loss/SRqqt2/detection_loss‚3.1388892171962652 epoch‚1 trainer/global_step‚449 _runtime‚ 307.933478772 _step‚36 2loss/SRqqt2/assignment_loss‚1.7630507499433872 1loss/FBt2/assignment_loss‚0.11041165119446682 /loss/FBt2/detection_loss‚20.056894191714896 0loss/FRt1/assignment_loss‚3.1204173388304532 %loss/total_loss‚41.72815038278236 /loss/FRt2/assignment_loss‚4.028909098408559 2loss/SRqqt1/assignment_loss‚1.4616626745314925 /loss/FRt1/detection_loss‚1.3659801868347343 /loss/FRt2/detection_loss‚0.3916557545176613 .loss/FBt1/detection_loss‚5.118257082556268 $ _timestamp‚1.784493151331872e+09$Þa{Ý)Ö‚(:ž æèôÒ‚ˆú¾) disk./.usagePercent‚76.55990543436914 disk.dm-1.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 2461.69140625 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 1109.6953125* proc.memory.percent‚17.624855041503906 cpu‚25.735752605150534% disk./.usageGB‚1175.3859481811523 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 2461.69140625$ disk.nvme1n1.out‚879.16845703125 proc.cpu.threads‚17 network.sent‚532657 network.recv‚128464" disk.dm-0.out‚3131.75341796875 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.dm-1.out‚ 192.98828125 disk.rbd0.in‚0& disk.nvme1c1n1.out‚879.16845703125 disk.md0.in‚0 disk.md0.out‚7.4375$ memory_percent‚93.22695732116699 proc.memory.rssMB‚ 2887.65625°d>3ž×‚(:“ æèôÒù‘Ü¿ gpu.0.gpu‚94.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚229.736' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.57866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù‰pœžØ‚(:“ õèôÒàñ¾ gpu.0.gpu‚62.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚221.434' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.81133333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G°‘5Ù‚(:ª õèôÒÂï­¿% disk./.usageGB‚1175.3895874023438 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2801.5400390625$ memory_percent‚93.22900772094727 cpu‚25.745033911179924 disk.dm-0.in‚0" disk.dm-0.out‚3471.47705078125$ disk.nvme0n1.out‚2801.5400390625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1354.59033203125# proc.memory.rssMB‚ 2887.66015625* proc.memory.percent‚17.624878883361816 network.sent‚596080 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚1354.59033203125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 7.76953125' proc.memory.availableMB‚ 1109.359375 network.recv‚133374) disk./.usagePercent‚76.56014247857638 disk.dm-1.out‚ 667.81640625 disk.nvme0n1.in‚0 proc.cpu.threads‚17„\._0Ú+ _wandbruntime‚340  _runtime‚340 ï¼ÄÛ¾ % _timestamp‚1.7844931839900572e+09 _runtime‚ 340.591925618 _step‚37 & lr-AdamW/pg1‚0.0001604401379296054 & lr-AdamW/pg2‚0.0001604401379296054 trainer/global_step‚499¤-\ÈÜ _runtime‚ 340.591925618 _step‚37 & lr-AdamW/pg1‚0.0001604401379296054 & lr-AdamW/pg2‚0.0001604401379296054 trainer/global_step‚499 % _timestamp‚1.7844931839900572e+09%þj0š0Ý+ _wandbruntime‚340  _runtime‚340ëB<µ'Þ¯O WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 trainer/global_step‚499 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 408549520021 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29339855418368 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213068800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 409590426112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8958579 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38137150353104 _runtime‚ 340.593055809 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5018306 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28917581405696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5008165 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1137342 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 422487081472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38545716912485 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38590489645568 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39000080071680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5008165 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409590426112 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940271 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118959104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38590506684928 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 409679085056 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3940273 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3940271 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39000080071680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38590489645568 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 409679085056 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3774725 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 408462733603 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5018306 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28917581405696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5008165 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1137342 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 422487081472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38545716912485 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38590489645568 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39000080071680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5008165 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 409590426112 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940271 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118959104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38590506684928 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 409679085056 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3940273 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3940271 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39000080071680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38590489645568 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 409679085056 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3774725 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 408462733603 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29339855418368 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213068800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 409590426112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8958579 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38137150353104 _runtime‚ 340.593055809 HDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3781953 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38620687106699 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5028140 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38620790932589 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 423295800320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 % _timestamp‚1.7844931844094403e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39076015453696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38665656351232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38211457917856 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410376141824 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3947902 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3947900 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39076121159168 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 409246228203 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _step‚39 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8976042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5028140 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 341.011496433 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 410376141824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28973625024000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1139507 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2642612 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8965885 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚162 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29396810260992 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 423185236992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚499 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39076121159168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38665639311872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3947902 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110563328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5017985 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112™Ìï Ã&â½M MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5028140 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38620790932589 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 423295800320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 % _timestamp‚1.7844931844094403e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39076015453696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38665656351232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38211457917856 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 410376141824 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3947902 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3947900 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39076121159168 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 409246228203 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _step‚39 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8976042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5028140 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 341.011496433 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 410376141824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28973625024000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1139507 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2642612 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8965885 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚162 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29396810260992 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 423185236992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚499 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39076121159168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38665639311872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3947902 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110563328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5017985 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3781953 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38620687106699 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812'ºKö0ã+ _wandbruntime‚341  _runtime‚341û—äû 1loss/SRqqt2/assignment_loss‚1.622897388733174 1loss/FBt1/assignment_loss‚0.19770983363409916 /loss/FRt1/detection_loss‚1.4361475814534905 _step‚40 1loss/SRqqt1/detection_loss‚1.0412214010466265 1loss/SRqqt2/detection_loss‚2.8629223195574514 % _timestamp‚1.7844931844105334e+09 epoch‚1 0loss/FRt1/assignment_loss‚3.0386162380827724 0loss/FBt2/assignment_loss‚0.2043376659948687 0loss/FRt2/detection_loss‚0.36356490068015523 &loss/total_loss‚30.409862304463473 2loss/SRqqt1/assignment_loss‚1.4813490623787597 .loss/FBt1/detection_loss‚4.574226201545504 .loss/FBt2/detection_loss‚9.510245185041667 trainer/global_step‚499 _runtime‚ 341.012091664 /loss/FRt2/assignment_loss‚4.076624526314904”8ÝŽåÿ .loss/FBt1/detection_loss‚4.574226201545504 .loss/FBt2/detection_loss‚9.510245185041667 trainer/global_step‚499 _runtime‚ 341.012091664 /loss/FRt2/assignment_loss‚4.076624526314904 1loss/SRqqt2/assignment_loss‚1.622897388733174 1loss/FBt1/assignment_loss‚0.19770983363409916 /loss/FRt1/detection_loss‚1.4361475814534905 _step‚40 1loss/SRqqt1/detection_loss‚1.0412214010466265 1loss/SRqqt2/detection_loss‚2.8629223195574514 % _timestamp‚1.7844931844105334e+09 epoch‚1 0loss/FRt1/assignment_loss‚3.0386162380827724 0loss/FBt2/assignment_loss‚0.2043376659948687 0loss/FRt2/detection_loss‚0.36356490068015523 &loss/total_loss‚30.409862304463473 2loss/SRqqt1/assignment_loss‚1.4813490623787597(\Eø4æ‚(:© „éôÒ‹ƒ°¿' proc.memory.availableMB‚ 1109.765625# proc.memory.rssMB‚ 2887.66015625 disk.rbd0.in‚0$ disk.nvme1n1.out‚1355.3935546875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0" disk.dm-0.out‚3476.07080078125$ memory_percent‚93.22652816772461* proc.memory.percent‚17.624878883361816 cpu‚25.737576609623893 network.sent‚612063 network.recv‚136149) disk./.usagePercent‚76.56017055613971 disk.dm-1.in‚0 disk.dm-1.out‚668.0732421875 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚1355.3935546875 disk.nvme1n1.in‚0% disk./.usageGB‚1175.3900184631348' disk.nvme0c0n1.out‚2806.42236328125% disk.nvme0n1.out‚2806.42236328125 disk.md0.in‚0 disk.md0.out‚8.15625 disk.dm-0.in‚0mjèç‚(:’ „éôÒžݿ gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.294' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.76466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Š¼Àœè‚(:‘ “éôҬɺ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.542' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.18066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜¤2é‚(:§ “éôÒݪâ¿ disk.md0.out‚ 8.52734375& proc.memory.availableMB‚ 1109.71875* proc.memory.percent‚17.624878883361816 network.sent‚661432) disk./.usagePercent‚76.56012607929159% disk./.usageGB‚1175.3893356323242 disk.rbd0.in‚0" disk.dm-0.out‚3486.21923828125 network.recv‚140713' disk.nvme0c0n1.out‚2816.14599609375 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2816.14599609375 disk.nvme1c1n1.in‚0" disk.nvme1n1.out‚ 1359.89453125$ memory_percent‚93.22681427001953 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚671.3662109375 disk.nvme0c0n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2887.66015625 cpu‚25.728660470964787 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 1359.89453125 disk.nvme1n1.in‚0IpÁV0ê+ _wandbruntime‚374  _runtime‚374(L* Äë¾ trainer/global_step‚549 % _timestamp‚1.7844932175659242e+09 _runtime‚ 374.167755619 _step‚41 & lr-AdamW/pg1‚0.0001604273600172034 & lr-AdamW/pg2‚0.0001604273600172034EŽêÈì _runtime‚ 374.167755619 _step‚41 & lr-AdamW/pg1‚0.0001604273600172034 & lr-AdamW/pg2‚0.0001604273600172034 trainer/global_step‚549 % _timestamp‚1.7844932175659242e+09)3Fß“0í+ _wandbruntime‚374  _runtime‚374 ÔJŽµ'î¯O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42347972960768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9832881 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42299377887115 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32187550488576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5501017 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5511158 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4321723 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _step‚42 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4321721 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42796913327104 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 449029025280 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449029025280 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4321721 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4135482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2890352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41851676622704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32187326934016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42347990000128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42347990000128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4135276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9822738 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1244924 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447788050829 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5511158 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448940366336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1245126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5501017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42796913327104!Ž+„/ ï³O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5511158 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448940366336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1245126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5501017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42796913327104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42347972960768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9832881 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42299377887115 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32187550488576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5501017 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5511158 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4321723 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _step‚42 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4321721 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42796913327104 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 449029025280 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449029025280 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4321721 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4135482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2890352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41851676622704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32187326934016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42347990000128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42347990000128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4135276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9822738 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1244924 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447788050829 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0*<˜Ì¿&ñ¹M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2895408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329350 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7844932179882166e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5510837 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1247254 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121049088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚549 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329352 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4142666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42872954414592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42872848709120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚43 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449726082048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 463771761664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 448571545541 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5520992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42423139666432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2895412 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31780511424000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31780605533696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41925967148096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42423139666432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5520992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 449814748160 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42872954414592 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42423122627072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 449726082048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32244498344448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32244283185664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4142500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215158784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9840187 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5510837 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 463892810752ˆIî´Ã&ò½M NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4142666 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42872954414592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42872848709120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚43 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449726082048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 463771761664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 448571545541 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5520992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42423139666432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2895412 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31780511424000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31780605533696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41925967148096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42423139666432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5520992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 449814748160 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42872954414592 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42423122627072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 449726082048 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32244498344448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32244283185664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4142500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215158784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9840187 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5510837 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 463892810752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2895408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329350 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7844932179882166e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5510837 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1247254 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121049088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚549 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329352 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155+“…·ã0ó+ _wandbruntime‚374  _runtime‚374*(ôú /loss/FBt2/detection_loss‚6.9197949870089195 %loss/total_loss‚28.50791081596181 0loss/FRt1/assignment_loss‚2.8111159156108725 /loss/FRt1/detection_loss‚1.3785659570401085 1loss/SRqqt1/detection_loss‚0.9476467898471748 trainer/global_step‚549 _step‚44 0loss/FRt2/assignment_loss‚3.9998955638921876 2loss/SRqqt1/assignment_loss‚1.3326068112422014 0loss/FBt1/assignment_loss‚0.2070435200250813 /loss/FBt2/assignment_loss‚0.145208098150489 /loss/FRt2/detection_loss‚0.4768642787231584 /loss/FBt1/detection_loss‚5.4831106429704874 % _timestamp‚1.7844932179893339e+09 2loss/SRqqt2/assignment_loss‚1.5886781389472653 0loss/SRqqt2/detection_loss‚3.217380112503865 epoch‚1 _runtime‚ 374.594267028AÀíõþ /loss/FBt2/detection_loss‚6.9197949870089195 %loss/total_loss‚28.50791081596181 0loss/FRt1/assignment_loss‚2.8111159156108725 /loss/FRt1/detection_loss‚1.3785659570401085 1loss/SRqqt1/detection_loss‚0.9476467898471748 trainer/global_step‚549 _step‚44 0loss/FRt2/assignment_loss‚3.9998955638921876 2loss/SRqqt1/assignment_loss‚1.3326068112422014 0loss/FBt1/assignment_loss‚0.2070435200250813 /loss/FBt2/assignment_loss‚0.145208098150489 /loss/FRt2/detection_loss‚0.4768642787231584 /loss/FBt1/detection_loss‚5.4831106429704874 % _timestamp‚1.7844932179893339e+09 2loss/SRqqt2/assignment_loss‚1.5886781389472653 0loss/SRqqt2/detection_loss‚3.217380112503865 epoch‚1 _runtime‚ 374.594267028,1«(F7ö‚(:¬ ¢éôÒåÍ÷¾% disk.nvme0n1.out‚2817.16357421875% disk.nvme1n1.out‚1360.74267578125 disk.dm-1.in‚0 network.recv‚144327 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% proc.memory.availableMB‚ 980.96875* proc.memory.percent‚17.624878883361816 cpu‚25.723168160277744 proc.cpu.threads‚17' disk.nvme1c1n1.out‚1360.74267578125 disk.md0.in‚0 disk.md0.out‚ 8.9921875 disk.dm-0.in‚0! disk.dm-1.out‚671.59912109375 disk.rbd0.out‚0$ memory_percent‚94.01264190673828# proc.memory.rssMB‚ 2887.66015625 network.sent‚666631% disk./.usageGB‚1175.3892974853516 disk.nvme1c1n1.in‚0" disk.dm-0.out‚3486.85595703125' disk.nvme0c0n1.out‚2817.16357421875) disk./.usagePercent‚76.56012359455147 disk.nvme0n1.in‚0lä ÷‚(:’ ¢éôÒ¨ÀØ¿ gpu.0.gpu‚55.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.508' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.16933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?´ú8ø‚(:­ ±éôÒ”ðê¾ disk.rbd0.out‚0# memory_percent‚94.0119743347168 cpu‚25.719226392743135 network.sent‚730744) disk./.usagePercent‚76.56011489796106 disk.dm-1.in‚0! disk.dm-1.out‚671.68115234375 disk.nvme0n1.in‚0% disk.nvme1n1.out‚1361.50341796875 disk.md0.in‚0 proc.cpu.threads‚17 network.recv‚149397% disk./.usageGB‚1175.3891639709473" disk.dm-0.out‚3489.93505859375 disk.rbd0.in‚0' disk.nvme1c1n1.out‚1361.50341796875& proc.memory.availableMB‚ 981.078125# proc.memory.rssMB‚ 2887.66015625 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2820.25732421875% disk.nvme0n1.out‚2820.25732421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 9.31640625* proc.memory.percent‚17.624878883361816 disk.dm-0.in‚0ߟó“ù‚(:ˆ ±éôÒù—š¿ gpu.0.gpu‚90.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚224.427' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚74.809 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`9_x4ú‚(:© ÀéôÒÊÆó¾ disk.dm-1.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚2821.06884765625& disk.nvme1c1n1.out‚1362.1904296875( proc.memory.availableMB‚ 981.07421875 network.recv‚152778 disk.md0.out‚ 9.5703125 disk.dm-1.out‚ 671.931640625 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme1n1.out‚1362.1904296875 cpu‚25.71494834478465 network.sent‚734163 disk.md0.in‚0" disk.dm-0.out‚3490.61474609375 disk.nvme0n1.in‚0# memory_percent‚94.0119981765747# proc.memory.rssMB‚ 2887.66015625* proc.memory.percent‚17.624878883361816 proc.cpu.threads‚17) disk./.usagePercent‚76.56010968000682% disk./.usageGB‚1175.3890838623047' disk.nvme0c0n1.out‚2821.06884765625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0 Ó×’bû‚(:‘ ÀéôÒ€£³¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.374' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.12466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.corr´/·†:ectedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù“q#0ü+ _wandbruntime‚407  _runtime‚407¦óêÃý½ & lr-AdamW/pg2‚0.0001604094308647346 trainer/global_step‚599 $ _timestamp‚1.784493250845125e+09 _runtime‚ 407.448268122 _step‚45 & lr-AdamW/pg1‚0.0001604094308647346m“ÝÇþÁ & lr-AdamW/pg2‚0.0001604094308647346 trainer/global_step‚599 $ _timestamp‚1.784493250845125e+09 _runtime‚ 407.448268122 _step‚45 & lr-AdamW/pg1‚0.0001604094308647346-vŽú˜0ÿ+ _wandbruntime‚407  _runtime‚407ëš›¹'€³O UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46593852280832 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 488290306560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 486939795219 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 488290306560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 503520713728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46105473315328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46053246513301 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46593746582528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703173 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1352172 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5993869 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46105456275968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45566219931664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34531354205696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34531260096000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _runtime‚ 407.448327022 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46593746582528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46105456275968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 % _timestamp‚1.7844932508458354e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 488378965504 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1351990 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45566202892304H(Z½'·O SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703173 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1352172 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5993869 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46105456275968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45566219931664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34531354205696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34531260096000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _runtime‚ 407.448327022 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46593746582528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46105456275968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 % _timestamp‚1.7844932508458354e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 488378965504 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1351990 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45566202892304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417271 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46593852280832 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 488290306560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 486939795219 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 488290306560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 503520713728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46105473315328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46053246513301 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46593746582528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0.² ž0‚+ _wandbruntime‚407  _runtime‚407(3½.Ã&ƒ½M N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4502238 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35091955759104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6003689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46669681964544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 487723289819 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4710802 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3148212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227741696 _runtime‚ 407.863449663 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10724646 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 489164688384 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 489076022272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 504330193408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46180622981632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46180605942272 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 489164688384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35091728017408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6013844 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6003689 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45640493417696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46128216707515 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487810076349 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1354030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10714489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 504463825408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 489076022272 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46128320533405 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46180605942272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960Q>-„ÁM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6013844 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6003689 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45640493417696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46128216707515 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487810076349 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 QEDeviceStat‘‹ ðšsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1354030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10714489 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 504463825408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 489076022272 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46128320533405 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46180605942272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4502238 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35091955759104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6003689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46669681964544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 487723289819 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4710802 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3148212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227741696 _runtime‚ 407.863449663 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10724646 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 489164688384 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 489076022272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 504330193408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46180622981632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46180605942272 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 489164688384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35091728017408/È<€Ð0…+ _wandbruntime‚407  _runtime‚407þc‹Ôÿ†ù &loss/total_loss‚31.671809705342525 epoch‚1 2loss/SRqqt1/assignment_loss‚1.3124972919123261 1loss/FBt2/assignment_loss‚0.19847687887019902 /loss/FBt1/detection_loss‚10.106066102305645 trainer/global_step‚599 /loss/FRt2/assignment_loss‚4.084427526974081 /loss/FRt2/detection_loss‚0.4302466598769456 1loss/SRqqt2/detection_loss‚3.0442391999645886 $ _timestamp‚1.784493251262938e+09 _runtime‚ 407.864549545 0loss/FRt1/assignment_loss‚2.6523403451060155 1loss/SRqqt2/assignment_loss‚1.536489324307279 0loss/FBt1/assignment_loss‚0.2158865055268008 1loss/SRqqt1/detection_loss‚0.8776998554274741 .loss/FBt2/detection_loss‚5.801038432236262 _step‚48 .loss/FRt1/detection_loss‚1.412401582834906§lí¦‡ý &loss/total_loss‚31.671809705342525 epoch‚1 2loss/SRqqt1/assignment_loss‚1.3124972919123261 1loss/FBt2/assignment_loss‚0.19847687887019902 /loss/FBt1/detection_loss‚10.106066102305645 trainer/global_step‚599 /loss/FRt2/assignment_loss‚4.084427526974081 /loss/FRt2/detection_loss‚0.4302466598769456 1loss/SRqqt2/detection_loss‚3.0442391999645886 $ _timestamp‚1.784493251262938e+09 _runtime‚ 407.864549545 0loss/FRt1/assignment_loss‚2.6523403451060155 1loss/SRqqt2/assignment_loss‚1.536489324307279 0loss/FBt1/assignment_loss‚0.2158865055268008 1loss/SRqqt1/detection_loss‚0.8776998554274741 .loss/FBt2/detection_loss‚5.801038432236262 _step‚48 .loss/FRt1/detection_loss‚1.4124015828349060C»øS4ˆ‚(:© ÏéôÒ†“™¿ disk.nvme1n1.in‚0 disk.dm-1.out‚ 672.052734375& disk.nvme1c1n1.out‚1363.0224609375$ disk.nvme1n1.out‚1363.0224609375" disk.dm-0.out‚3497.01611328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.in‚0 network.recv‚156691% disk./.usageGB‚1175.3892059326172 disk.nvme0n1.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 980.9609375* proc.memory.percent‚17.624878883361816 cpu‚25.709785692552792) disk./.usagePercent‚76.56011763117519 disk.nvme0c0n1.in‚0 disk.md0.out‚ 9.8984375 disk.dm-0.in‚0# memory_percent‚94.0126895904541# proc.memory.rssMB‚ 2887.66015625 proc.cpu.threads‚17 network.sent‚752196' disk.nvme0c0n1.out‚2827.46240234375% disk.nvme0n1.out‚2827.46240234375 disk.nvme1c1n1.in‚0cC™L“‰‚(:ˆ ÏéôÒ§óË¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚233.313' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚77.771 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´aé&Š‚(:› ÞéôÒÙÓè¾" disk.nvme1c1n1.out‚ 1363.546875 disk.dm-0.in‚0" disk.dm-0.out‚3497.66064453125 disk.rbd0.in‚0 disk.nvme0n1.in‚0% disk./.usageGB‚1175.3892707824707 disk.nvme1c1n1.in‚0 disk.md0.out‚10.125 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2828.26416015625$ proc.memory.availableMB‚980.9375* proc.memory.percent‚17.624878883361816 proc.cpu.threads‚17 network.sent‚801637( disk./.usagePercent‚76.5601218552334 disk.nvme1n1.out‚ 1363.546875 disk.md0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2887.66015625 cpu‚25.705605715328264 network.recv‚161203 disk.nvme1n1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚2828.26416015625$ memory_percent‚94.01283264160156 disk.dm-1.out‚ 672.216796875ºâaÀž‹‚(:“ ÞéôÒ‰þ«¿ gpu.0.gpu‚65.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚115.153' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.38433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|àí0Œ+ _wandbruntime‚441  _runtime‚441ãm…8Å¿ _step‚49 ' lr-AdamW/pg1‚0.00016038635162375498 ' lr-AdamW/pg2‚0.00016038635162375498 trainer/global_step‚649 $ _timestamp‚1.784493284411997e+09 _runtime‚ 441.013828158,¾ŸÆÉŽÃ $ _timestamp‚1.784493284411997e+09 _runtime‚ 441.013828158 _step‚49 ' lr-AdamW/pg1‚0.00016038635162375498 ' lr-AdamW/pg2‚0.00016038635162375498 trainer/global_step‚6491Óýf¡0+ _wandbruntime‚441  _runtime‚441½ÞÏݹ'³O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 544088303104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49280746201264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49862939591168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221457408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49862956630528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526178326027 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6486721 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11581485 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496862 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527640246784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11571342 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚49862956630528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 544215650816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1458900 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49807011313709 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚649 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50390579837952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49862939591168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37338146496000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚50 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496862 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37338240605696 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49806907487931 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4855022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527728905728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3395956 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527640246784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚170 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3395952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5084621 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50390685536256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37882234799104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49280729161904!w¿- &‘·O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496862 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527640246784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11571342 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚49862956630528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 544215650816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1458900 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49807011313709 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚649 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50390579837952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49862939591168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37338146496000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚50 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496862 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37338240605696 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49806907487931 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4855022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527728905728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3395956 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527640246784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚170 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3395952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5084621 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50390685536256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37882234799104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49280729161904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 544088303104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49280746201264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49862939591168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221457408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49862956630528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526178326027 K?DeviceStatsMonitor.on_train_batch_start ÷´ /active.small_pool.freed‚6486721 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚115814852Ü#…j0’+ _wandbruntime‚441  _runtime‚441Ys6ªÃ&“½M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528514628608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6496541 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50466515219968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1460972 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5092250 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6496541 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217255936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49355036726656 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527048607157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4862151 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544901747712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 526961820627 _runtime‚ 441.433988836 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37939185971712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4861980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3401012 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37394284224000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6506696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50466620925440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49355019687296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528425962496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0‘5CÒÇ&”ÁM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4862151 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544901747712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 526961820627 _runtime‚ 441.433988836 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37939185971712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4861980 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3401012 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37394284224000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6506696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50466620925440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49355019687296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528425962496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528514628608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6496541 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50466515219968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1460972 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5092250 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6496541 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217255936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49355036726656 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527048607157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚03¦›¯0•+ _wandbruntime‚441  _runtime‚441«¢LÑÿ–ù 2loss/SRqqt1/assignment_loss‚1.3149810652600953 1loss/SRqqt2/assignment_loss‚1.632549356981846 1loss/FBt1/assignment_loss‚0.23373845133723162 .loss/FBt2/detection_loss‚10.42030793011954 &loss/total_loss‚27.803597157377993 0loss/FRt1/assignment_loss‚2.5908158368014353 epoch‚1 trainer/global_step‚649 % _timestamp‚1.7844932848333673e+09 /loss/FRt1/detection_loss‚1.4048490517146655 /loss/FRt2/detection_loss‚0.5229534102568563 1loss/SRqqt2/detection_loss‚2.7109988278601382 /loss/FRt2/assignment_loss‚4.026420301017339 0loss/FBt2/assignment_loss‚0.3051197307475718 1loss/SRqqt1/detection_loss‚0.9030493551712336 -loss/FBt1/detection_loss‚1.73781384011004 _runtime‚ 441.435037847 _step‚52¿v½\—ý % _timestamp‚1.7844932848333673e+09 /loss/FRt1/detection_loss‚1.4048490517146655 /loss/FRt2/detection_loss‚0.5229534102568563 1loss/SRqqt2/detection_loss‚2.7109988278601382 /loss/FRt2/assignment_loss‚4.026420301017339 0loss/FBt2/assignment_loss‚0.3051197307475718 1loss/SRqqt1/detection_loss‚0.9030493551712336 -loss/FBt1/detection_loss‚1.73781384011004 _runtime‚ 441.435037847 _step‚52 2loss/SRqqt1/assignment_loss‚1.3149810652600953 1loss/SRqqt2/assignment_loss‚1.632549356981846 1loss/FBt1/assignment_loss‚0.23373845133723162 .loss/FBt2/detection_loss‚10.42030793011954 &loss/total_loss‚27.803597157377993 0loss/FRt1/assignment_loss‚2.5908158368014353 epoch‚1 trainer/global_step‚6494ö‘mx1˜‚(:¦ íéôÒäà—¿# disk.nvme1n1.out‚1364.326171875 disk.md0.out‚ 10.515625" disk.dm-0.out‚3499.25537109375) disk./.usagePercent‚76.55923331216732 disk.nvme1n1.in‚0 disk.dm-1.out‚ 672.330078125% disk.nvme0n1.out‚2830.04052734375 proc.cpu.threads‚17 network.sent‚820690 network.recv‚165081% disk./.usageGB‚1175.3756294250488 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0" memory_percent‚94.014573097229% disk.nvme1c1n1.out‚1364.326171875 disk.md0.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚2830.04052734375( proc.memory.availableMB‚ 980.65234375# proc.memory.rssMB‚ 2887.66015625* proc.memory.percent‚17.624878883361816 cpu‚25.70053152163418 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0{ýê‘œ™‚(:‘ íéôÒ£ Æ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.767' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.92233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÛsŒ8š‚(:­ üéôÒä›ñ¾$ disk.nvme0n1.out‚2831.0439453125 disk.dm-0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2887.66015625) disk./.usagePercent‚76.55924399654982 disk.rbd0.in‚0' disk.nvme1c1n1.out‚1365.05908203125 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-0.out‚3499.84130859375 disk.dm-1.out‚672.4423828125& disk.nvme0c0n1.out‚2831.0439453125 disk.md0.out‚ 11.00390625$ memory_percent‚94.01495456695557( proc.memory.availableMB‚ 980.58984375 cpu‚25.69793863668594 proc.cpu.threads‚17 network.recv‚169592% disk./.usageGB‚1175.3757934570312 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1365.05908203125* proc.memory.percent‚17.624878883361816 network.sent‚869235 ¹›‚(:’ üéôÒÒü¤¿ gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚222.266' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.08866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U.¬ñ0œ+ _wandbruntime‚474  _runtime‚474ÞTÿZÆÀ trainer/global_step‚699 % _timestamp‚1.7844933174795563e+09 _runtime‚ 474.081495616 _step‚53 ' lr-AdamW/pg1‚0.00016035812377660101 ' lr-AdamW/pg2‚0.00016035812377660101eŽíœÊžÄ % _timestamp‚1.7844933174795563e+09 _runtime‚ 474.081495616 _step‚53 ' lr-AdamW/pg1‚0.00016035812377660101 ' lr-AdamW/pg2‚0.00016035812377660101 trainer/global_step‚6995ú3'J0Ÿ+ _wandbruntime‚474  _runtime‚474ŸŽ2õ ²O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1565132 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 584620925952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54187518791680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5214063 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54187413093376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 565503643253 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 565416856835 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466073 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3648752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223554560 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40729653821952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53560776114117 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53620439945728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚699 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6979573 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6979573 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12445644 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40145127005696 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6989714 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53620422906368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 566990187008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53560672288339 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53620439945728 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 567078845952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52995255431504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 474.085100882 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6989714 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53620422906368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 566990187008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53560672288339 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53620439945728 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 567078845952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52995255431504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 474.085100882 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 584620925952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54187518791680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466071 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5214063 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54187413093376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 565503643253 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 565416856835 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466073 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 2692997126ß¿Ã0¢+ _wandbruntime‚474  _runtime‚474ŽþR‰Ã&£½M F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 567775902720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54263348475392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53695572572672 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 566287137965 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5473700 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40786822864896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219353088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚699 % _timestamp‚1.7844933178872216e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53069562996256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚55 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5473702 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53635746308331 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54263454180864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53069545956896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53695589612032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6989393 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5221059 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1567251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 567864568832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53695589612032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5473700 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5221230 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12463093 _runtime‚ 474.489284292 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54263454180864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5473702 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 585432887808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3653812 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53695572572672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 567775902720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 585558131200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 567864568832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12473250 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40201170624000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 143654912007°aÇ&¤ÁM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3653808 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6999548 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1567418 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125243392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53635850134221 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53695589612032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6989393 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5221059 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1567251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 567864568832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53695589612032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5473700 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5221230 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12463093 _runtime‚ 474.489284292 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54263454180864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5473702 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 585432887808 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3653812 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53695572572672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 567775902720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 585558131200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 567864568832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12473250 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40201170624000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 567775902720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54263348475392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53695572572672 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 566287137965 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5473700 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40786822864896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219353088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚699 % _timestamp‚1.7844933178872216e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53069562996256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚55 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5473702 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53635746308331 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54263454180864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53069545956896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57377906221568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 604742174061 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43577273951232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚58 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57984352047104 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7472425 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42951919296000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5847523 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1671640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5847521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 625131100672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57984246348800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 606428786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5847521 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56709781701104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3901556 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223554560 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57314540914525 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57377906221568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1671468 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 625260545536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57984352047104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57377923260928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 604655387643 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 trainer/global_step‚749 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 606340127232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42952013405696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57984246348800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 % _timestamp‚1.7844933489492943e+09 _runtime‚ 505.5510594 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57377923260928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43577050396672 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¯²Á»'±µO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 606428786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88658944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5847521 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56709781701104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3901556 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223554560 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57314540914525 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57377906221568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1671468 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 625260545536 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57984352047104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57377923260928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 604655387643 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 trainer/global_step‚749 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 606340127232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42952013405696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57984246348800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 % _timestamp‚1.7844933489492943e+09 _runtime‚ 505.5510594 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57377923260928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43577050396672 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7472425 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13319946 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 606428786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88658944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5573020 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57314437088747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7482566 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 606340127232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825778 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3901552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129444864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57377906221568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 604742174061 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43577273951232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚58 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57984352047104 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7472425 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42951919296000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5847523 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1671640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5847521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 625131100672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57984246348800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0:ùRr0²+ _wandbruntime‚505  _runtime‚505=s‘Aá³½M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3906608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13337395 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7482245 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5855150 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43633996441088 _runtime‚ 505.953193846 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 607214509056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5855152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57453055887872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43634222085632 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 625939417088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56784072226496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 % _timestamp‚1.7844933493512077e+09 _step‚59 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57453072927232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58060181730816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 FDeviceStatsMonitor.on_train_batch_end/requêÿâested_bytes.all.peak‚ 12106884167 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58060287436288 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5580116 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 607125842944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5580306 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 626070951936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131534848 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5855152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5855150 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7482245)ÀìÇ&´ÁM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825890 trainer/global_step‚749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225644544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43008151133696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269299712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57453055887872 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7492400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58060287436288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7492400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57389511108739 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1673508 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58060287436288 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5580116 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 607125842944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5580306 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 626070951936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131534848 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5855152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5855150 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7482245 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3906608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13337395 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7482245 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5855150 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43633996441088 _runtime‚ 505.953193846 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 607214509056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5855152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57453055887872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786530 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43634222085632 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 625939417088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56784072226496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 % _timestamp‚1.7844933493512077e+09 _step‚59 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57453072927232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58060181730816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1778300 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 665630360064 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 645690067456 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46424653319168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚160 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 665753513472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61135406576128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60424307970704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7975418 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7965277 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61068305714933 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61135389536768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123153408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 643980704869 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61781185302528 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6228973 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6228971 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1778460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7965277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61135389536768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61068201889155 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61781185302528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217263104 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 643893918451 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61781079604224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45758805696000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4154356 _step‚62 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7975418 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60424325010064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61781079604224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46424436056064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45758899805696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6228973 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5932816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 645778726400 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5932652 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Æ,ŽZÁ·O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7965277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61135389536768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61068201889155 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884167 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2370611712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61781185302528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217263104 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 643893918451 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61781079604224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45758805696000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4154356 _step‚62 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7975418 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60424325010064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61781079604224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46424436056064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45758899805696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6228973 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5932816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 645778726400 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5932652 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269299712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6228971 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 645690067456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14194248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1778300 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 665630360064 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 645690067456 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46424653319168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QIDeviceStatsMonitª(cor.on_train_batch_start/inactive_split.small_pool.current‚160 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 665753513472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61135406576128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60424307970704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7975418 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7965277 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61068305714933 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61135389536768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269299712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123153408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 643980704869 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61781185302528 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6228973 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6228971 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1778460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486>FY”‹0Â+ _wandbruntime‚537  _runtime‚53775èÃ&ýM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705472 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14211697 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60498615535456 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 644677413051 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61210539203072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127340544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61210539203072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6236602 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4159412 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1780288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 646475783168 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6236600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1780457 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 _step‚63 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14221854 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45814943424000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61857120691712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61210556242432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚169 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7985252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 % _timestamp‚1.7844933807811382e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 666436661248 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61143379735037 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61143275909147 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7985252 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5939696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45815037533696 _runtime‚ 537.383120739 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221450240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7975097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46481380085248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46481601535488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60498598496096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7975097 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7985252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 % _timestamp‚1.7844933807811382e+09 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 666436661248 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61143379735037 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61143275909147 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417271 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7985252 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5939696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45815037533696 _runtime‚ 537.383120739 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221450240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7975097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46481380085248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46481601535488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60498598496096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7975097 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127340544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61210539203072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269299712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6236602 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4159412 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1780288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 646475783168 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6236600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1780457 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2370611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 _step‚63 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14221854 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45814943424000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884167 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61857120691712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61210556242432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚169 B&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9541216492652893 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1563827395439148 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3820129930973053 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8269230723381042 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_2_of_4‚0.16775095462799072 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9948750734329224 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.25866520404815674 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.06861988455057144 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2781588137149811 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2752376198768616 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28815245628356934 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 epoch‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938583374023438 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8981481194496155 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0077300299890339375 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009999999776482582 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9033105969429016 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.060211874544620514 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.31606197357177734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012368584284558892 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.253438800573349 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33346426486968994 +particle/sensitivity‚0.5279399156570435 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5400312542915344 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9448484778404236 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05532347038388252 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.289784699678421 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.850873589515686 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.3656270503997803 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.021899614483118057 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6146748661994934 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 'particle/f_score‚0.5385768413543701 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784493393912754e+09 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15434397757053375 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1682967096567154 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9767107963562012 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9516553282737732 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9804688692092896 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2844761312007904 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9945418834686279 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.28610026836395264 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.254526287317276 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_3‚0.707629919052124 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3989418148994446 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 , Purity/2FRt*SR”'5,;&qqt2FBt/FRt_purity‚0.09375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0037029345985502005 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3603525757789612 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2540999948978424 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07402829080820084 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.20499999821186066 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9808759093284607 jet/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6990740299224854 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2890625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.6677151322364807 _runtime‚ 550.515177588 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,jet/accuracy_3_of_4‚0.024073466658592224 particle/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.26401275396347046 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4277589023113251 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.27984195947647095 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.35893669724464417 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8048751354217529 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9931281208992004 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12942883372306824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23030169308185577 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.2995087206363678 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _step‚65 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9931546449661255 *jet/accuracy_2_of_2‚0.0675179585814476 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01120905764400959 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.25989869236946106 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 +particle/specificity‚0.8816954493522644 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.28902187943458557 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9931546449661255 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_4‚0.9970238208770752 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07358887046575546 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3868027627468109 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13792087137699127 (particle/accuracy‚0.8057259917259216 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.25505661964416504 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.29740849137306213 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2844761312007904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.07298587262630463 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6959296464920044 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31009212136268616 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07989246398210526 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43204838037490845 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16302597522735596 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3831571936607361 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.282070517539978 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7435348033905029 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24722817540168762 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6990740299224854 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33385521173477173 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009463159367442131 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28727632761001587 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009631956927478313 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.35404953360557556 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1467110514640808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2040420025587082 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0024420025292783976 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.24939745664596558 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2239583432674408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.36357349157333374 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3604583144187927 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07402829080820084 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.001760353334248066 trainer/global_step‚799 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10625926405191422 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5083840489387512 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16037896275520325 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.022493772208690643 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46364325284957886 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,jet/accuracy_3_of_3‚0.025508806109428406 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9839262962341309 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.27588045597076416 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2665245532989502 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0357142873108387 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 /particle/accuracy_4_of_4‚0.9348635077476501 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.96002197265625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5658875107765198 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8118039965629578 *loss/val_total_loss‚21.846209146647624 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951454997062683 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2574106752872467 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9920566082000732 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.29002150893211365 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.28212693333625793 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2594905495643616 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.24939745664596558 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2543155252933502 jet/accuracy_4_of_4‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994846522808075 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5352001786231995 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5970044136047363 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.26552051305770874 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07091923803091049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08261149376630783 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.25374266505241394 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.5726551413536072 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6243517994880676 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12942883372306824 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23209957778453827 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2222222238779068 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.987610936164856 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2709241509437561 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9755951762199402 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2523035407066345"vu0š=Ì”{ 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.96002197265625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5658875107765198 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8118039965629578 *loss/val_total_loss‚21.846209146647624 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9951454997062683 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2574106752872467 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9920566082000732 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.29002150893211365 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.28212693333625793 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2594905495643616 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.24939745664596558 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2543155252933502 jet/accuracy_4_of_4‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994846522808075 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5352001786231995 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5970044136047363 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.26552051305770874 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07091923803091049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08261149376630783 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.25374266505241394 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.5726551413536072 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6243517994880676 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12942883372306824 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23209957778453827 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2222222238779068 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.987610936164856 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2709241509437561 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9755951762199402 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2523035407066345 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08178883045911789 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08962379395961761 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13964694738388062 +jet/accuracy_1_of_1‚0.16126663982868195 *jet/accuracy_2_of_3‚0.2197948694229126 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1647106409072876 7validation_average_jet_accuracy‚0.20574668049812317 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9850245118141174 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.14487668871879578 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2986111044883728 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9825333952903748 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9541216492652893 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1563827395439148 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3820129930973053 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8269230723381042 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_2_of_4‚0.16775095462799072 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9948750734329224 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.25866520404815674 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.06861988455057144 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2781588137149811 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2752376198768616 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28815245628356934 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 epoch‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938583374023438 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8981481194496155 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0077300299890339375 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009999999776482582 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9033105969429016 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.060211874544620514 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.31606197357177734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012368584284558892 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.253438800573349 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.33346426486968994 +particle/sensitivity‚0.5279399156570435 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5400312542915344 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9448484778404236 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05532347038388252 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.289784699678421 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.850873589515686 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.3656270503997803 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.021899614483118057 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6146748661994934 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 'particle/f_score‚0.5385768413543701 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784493393912754e+09 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15434397757053375 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1682967096567154 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9767107963562012 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9516553282737732 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9804688692092896 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2844761312007904 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9945418834686279 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.28610026836395264 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.254526287317276 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_3‚0.707629919052124 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3989418148994446 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.09375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0037029345985502005 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3603525757789612 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2540999948978424 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.07402829080820084 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.20499999821186066 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9808759093284607 jet/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6990740299224854 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2890625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.6677151322364807 _runtime‚ 550.515177588 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,jet/accuracy_3_of_4‚0.024073466658592224 particle/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.26401275396347046 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4277589023113251 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.27984195947647095 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.35893669724464417 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8048751354217529 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9931281208992004 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12942883372306824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23030169308185577 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.2995087206363678 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _step‚65 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9931546449661255 *jet/accuracy_2_of_2‚0.0675179585814476 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01120905764400959 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.25989869236946106 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 +particle/specificity‚0.8816954493522644 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.28902187943458557 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9931546449661255 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_4‚0.9970238208770752 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07358887046575546 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3868027627468109 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13792087137699127 (particle/accuracy‚0.8057259917259216 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.25505661964416504 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.29740849137306213 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2844761312007904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.07298587262630463 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6959296464920044 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31009212136268616 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07989246398210526 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43204838037490845 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16302597522735596 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3831571936607361 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.282070517539978 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7435348033905029 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24722817540168762 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6990740299224854 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.33385521173477173 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009463159367442131 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.28727632761001587 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009631956927478313 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.35404953360557556 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1467110514640808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2040420025587082 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0024420025292783976 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.24939745664596558 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2239583432674408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.36357349157333374 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3604583144187927 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.07402829080820084 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.001760353334248066 trainer/global_step‚799 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10625926405191422 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5083840489387512 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16037896275520325 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.022493772208690643 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46364325284957886 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,jet/accuracy_3_of_3‚0.025508806109428406 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9839262962341309 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.27588045597076416 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2665245532989502 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0357142873108387 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 /particle/accuracy_4_of_4‚0.9348635077476501AóKj’žÍ‚(:“ ÖêôÒéö¹¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚223.223' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.40766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰‹Ç7΂(:¬ ÖêôÒè˜ç¿! disk.dm-0.out‚3517.3408203125 cpu‚25.784747869838537 proc.cpu.threads‚17 network.sent‚ 1.071802e+06 disk.dm-1.out‚737.0166015625 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚94.45183277130127( proc.memory.availableMB‚ 909.01171875 network.recv‚195052$ disk.nvme0n1.out‚2849.4970703125 disk.dm-0.in‚0* proc.memory.percent‚17.667365074157715& disk.nvme1c1n1.out‚1433.2587890625) disk./.usagePercent‚76.56142112584091 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2894.62109375% disk./.usageGB‚1175.4092178344727 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2849.4970703125$ disk.nvme1n1.out‚1433.2587890625 disk.md0.out‚13.125ûÂ>`5Ï‚(:ª åêôÒΛö¾% disk.nvme0c0n1.out‚2851.373046875 disk.nvme0n1.in‚0# disk.nvme0n1.out‚2851.373046875 network.recv‚199836) disk./.usagePercent‚76.56141615636068 disk.md0.in‚0 disk.dm-0.out‚ 3519.203125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚94.45669651031494( proc.memory.availableMB‚ 908.21484375% disk./.usageGB‚1175.4091415405273$ disk.nvme1n1.out‚1433.9560546875 disk.dm-1.out‚737.0751953125# proc.memory.rssMB‚ 2895.61328125* proc.memory.percent‚17.673420906066895 proc.cpu.threads‚17& disk.nvme1c1n1.out‚1433.9560546875 disk.nvme1n1.in‚0 disk.md0.out‚ 13.42578125 disk.rbd0.out‚0 cpu‚25.78051143929239 network.sent‚ 1.130815e+06 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0?E#9žÐ‚(:“ åêôÒ“Ô¤¿ gpu.0.gpu‚82.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚220.442' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.48066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÖ\Ž0Ñ+ _wandbruntime‚582  _runtime‚582½·‡ÆÒÀ ' lr-AdamW/pg2‚0.00016024256838101885 trainer/global_step‚849 % _timestamp‚1.7844934255220594e+09 _runtime‚ 582.123915646 _step‚66 ' lr-AdamW/pg1‚0.00016024256838101885K?-ÊÓÄ _step‚66 ' lr-AdamW/pg1‚0.00016024256838101885 ' lr-AdamW/pg2‚0.00016024256838101885 trainer/global_step‚849 % _timestamp‚1.7844934255220594e+09 _runtime‚ 582.123915646BÊö°™0Ô+ _wandbruntime‚582  _runtime‚582ú ¢ÍFÕ±O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6732221 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66589935228928 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 700469382144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6447411 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 698547707817 trainer/global_step‚849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65810848984313 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721796801024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66589829530112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215165440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66589935228928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 # _timestamp‚1.78449342552295e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65889377187328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_star#:Cqt/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65889360147968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6732223 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49369151410688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65112301276496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8550773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚67 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6732221 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8540631 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66589829530112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 _runtime‚ 582.128263354 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65889360147968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49369057300992 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 721917856768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65810952810095 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4515411 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65112318315856 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15272852 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65889377187328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4515407 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6732223 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8550773 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760e¦Vš»'ÖµO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65810848984313 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721796801024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66589829530112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215165440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66589935228928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 # _timestamp‚1.78449342552295e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65889377187328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65889360147968 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6732223 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49369151410688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65112301276496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8550773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚67 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6732221 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8540631 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66589829530112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 _runtime‚ 582.128263354 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65889360147968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49369057300992 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 721917856768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65810952810095 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4515411 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65112318315856 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15272852 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65889377187328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4515407 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6732223 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8550773 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6732221 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66589935228928 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 700469382144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6447411 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 698547707817 trainer/global_step‚849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 701255097856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8560607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8550451 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6739850 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 722729614848 $ _timestamp‚1.784493425925125e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4520463 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 701343764480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6739850 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8550451 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49425289138688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66665870618112 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66665764912128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 722608566272 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65964526853632 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6454794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 582.527152039 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8560607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15300459 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65186608841248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6739852 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚849 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65186591801888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _step‚68 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 701255097856 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65964509814272 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6454620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65964526853632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65886026830199 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65885923004305 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215158272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49425195028992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15290301 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 699417988951 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1934327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Ó`¿ÙÀM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65186591801888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _step‚68 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 701255097856 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65964509814272 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6454620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65964526853632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65886026830199 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65885923004305 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215158272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49425195028992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15290301 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 699417988951 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1934327 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 701343764480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6739852 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65964509814272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50147803595264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 701255097856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8560607 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8550451 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6739850 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 722729614848 $ _timestamp‚1.784493425925125e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4520463 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 701343764480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6739850 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8550451 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49425289138688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66665870618112 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66665764912128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 722608566272 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.a+BEß llocated‚65964526853632 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6454794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 582.527152039 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8560607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15300459 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65186608841248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6739852 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚849 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0DœýÈ0Ú+ _wandbruntime‚582  _runtime‚582Aç—UþÛø /loss/FRt2/assignment_loss‚3.912855637798334 1loss/SRqqt1/assignment_loss‚1.159955581992644 &loss/total_loss‚25.479569984448517 % _timestamp‚1.7844934259260564e+09 0loss/FRt1/assignment_loss‚2.2836166182119304 2loss/SRqqt2/assignment_loss‚1.4936198401492793 0loss/FBt1/assignment_loss‚0.1337589025084197 1loss/FBt2/assignment_loss‚0.18961625799877202 1loss/SRqqt1/detection_loss‚0.8453025209926648 /loss/FRt2/detection_loss‚0.5121213346250617 /loss/SRqqt2/detection_loss‚2.02335782329143 .loss/FBt1/detection_loss‚7.907984864344497 /loss/FBt2/detection_loss‚3.5468733896052473 epoch‚2 _runtime‚ 582.52763174 _step‚69 /loss/FRt1/detection_loss‚1.4705072129302372 trainer/global_step‚849&|(dÜü 2loss/SRqqt2/assignment_loss‚1.4936198401492793 0loss/FBt1/assignment_loss‚0.1337589025084197 1loss/FBt2/assignment_loss‚0.18961625799877202 1loss/SRqqt1/detection_loss‚0.8453025209926648 /loss/FRt2/detection_loss‚0.5121213346250617 /loss/SRqqt2/detection_loss‚2.02335782329143 .loss/FBt1/detection_loss‚7.907984864344497 /loss/FBt2/detection_loss‚3.5468733896052473 epoch‚2 _runtime‚ 582.52763174 _step‚69 /loss/FRt1/detection_loss‚1.4705072129302372 trainer/global_step‚849 /loss/FRt2/assignment_loss‚3.912855637798334 1loss/SRqqt1/assignment_loss‚1.159955581992644 &loss/total_loss‚25.479569984448517 % _timestamp‚1.7844934259260564e+09 0loss/FRt1/assignment_loss‚2.2836166182119304E¤Œ¬+Ý‚(:  ôêôÒû¯Í¾ disk.rbd0.in‚0& disk.nvme0c0n1.out‚2852.2021484375 disk.nvme1c1n1.in‚0" disk.nvme1n1.out‚ 1498.43359375 disk.dm-0.out‚ 3519.78125# proc.memory.availableMB‚907.875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚2852.2021484375 disk.nvme1n1.in‚0 disk.md0.out‚ 13.7578125$ memory_percent‚94.45877075195312" proc.memory.rssMB‚ 2895.6171875* proc.memory.percent‚17.673444747924805 disk.md0.in‚0 cpu‚25.776854232633234 proc.cpu.threads‚17 network.sent‚ 1.134175e+06 network.recv‚203428 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 1498.43359375 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚801.0751953125) disk./.usagePercent‚76.56141764720475$ disk./.usageGB‚1175.409164428711 ~úù’Þ‚(:‡ ôêôÒ”ƒô¾ gpu.0.gpu‚54.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚223.221' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚74.407 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃEH÷5ß‚(:ª ƒëôÒÁä‘¿ disk.dm-0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2854.47314453125 disk.nvme0n1.in‚0% disk.nvme0n1.out‚2854.47314453125 disk.nvme1c1n1.in‚0 network.recv‚209627 cpu‚25.7728931356784 proc.cpu.threads‚17 disk.md0.in‚0" disk.dm-0.out‚3521.95849609375# disk.nvme1c1n1.out‚ 1499.3359375$ memory_percent‚94.46032047271729( proc.memory.availableMB‚ 907.62109375" proc.memory.rssMB‚ 2895.6171875* proc.memory.percent‚17.673444747924805 network.sent‚ 1.197156e+06( disk./.usagePercent‚76.5614293254833 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 1499.3359375 disk.dm-1.in‚0 disk.dm-1.out‚801.2197265625 disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1175.4093437194824 disk.md0.out‚ 14.16015625ׂ³‹žà‚(:“ ƒëôÒ锑¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚221.753' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.91766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ÷ñ0á+ _wandbruntime‚613  _runtime‚613 uûiÄâ¾ _runtime‚ 613.719625983 _step‚70 & lr-AdamW/pg1‚0.0001601937675443658 & lr-AdamW/pg2‚0.0001601937675443658 trainer/global_step‚899 % _timestamp‚1.7844934571178594e+09$ãE$Èã trainer/global_step‚899 % _timestamp‚1.7844934571178594e+09 _runtime‚ 613.719625983 _step‚70 & lr-AdamW/pg1‚0.0001601937675443658 & lr-AdamW/pg2‚0.0001601937675443658F<× 0ä+ _wandbruntime‚613  _runtime‚613à€Ö¸'å²O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70386768484352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70386662785536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52176037810688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4768207 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68826827546096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70386662785536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6809470 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 trainer/global_step‚899 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52938620824576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69646860502528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4768211 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9043625 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809295 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 739819322368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52175943700992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9033483 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 739907981824 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 762583013888 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762457763840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69564613784721 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 739819322368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 737786238625 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9033483 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52938401464832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130±zêX¼'æ¶O IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70386662785536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6809470 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 trainer/global_step‚899 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113671 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52938620824576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69646860502528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4768211 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9043625 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809295 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 739819322368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52175943700992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9033483 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 739907981824 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 762583013888 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762457763840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69564613784721 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 739819322368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 737786238625 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9033483 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52938401464832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70386768484352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70386662785536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52176037810688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4768207 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68826827546096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373G{ÔÌe0ç+ _wandbruntime‚614  _runtime‚614tá×e9è½M JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52232175538688 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69639687804713 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 763386186752 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 763271429632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9043303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69721993129472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16164603 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816714 trainer/global_step‚899 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52995352858624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚81‚Mn6Š2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69639791630607 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 614.124277799 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 740693704704 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 738656519759 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16174761 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16164603 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 740605038080 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70462703873536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70462598167552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9053459 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 740693704704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70462703873536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2043273 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52995561725440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68901118071488 _step‚72 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121300 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4773267 % _timestamp‚1.7844934575222826e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7121302 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4773263 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52232175538688 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69639687804713 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 763386186752 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 763271429632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9043303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69721993129472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16164603 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816714 trainer/global_step‚899 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52995352858624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69639791630607 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88666624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 614.124277799 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 740693704704 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 738656519759 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16174761 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16164603 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 740605038080 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70462703873536 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70462598167552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9053459 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 740693704704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70462703873536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2043273 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52995561725440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68901118071488 _step‚72 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121300 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4773267 % _timestamp‚1.7844934575222826e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7121302 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4773263 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682Ha£ 0ê+ _wandbruntime‚614  _runtime‚614û[ëú /loss/FRt2/detection_loss‚0.5976629577184498 1loss/SRqqt2/detection_loss‚2.3796535588514574 _runtime‚ 614.12475215 _step‚73 1loss/SRqqt1/assignment_loss‚1.138598061792476 0loss/SRqqt1/detection_loss‚0.839197559505646 epoch‚2 /loss/FRt1/assignment_loss‚2.286610598445462 1loss/FBt1/assignment_loss‚0.19199671171608088 1loss/FBt2/assignment_loss‚0.22426784951696954 .loss/FBt1/detection_loss‚2.793809578119745 /loss/FBt2/detection_loss‚3.1479473655227546 &loss/total_loss‚20.384920628477683 trainer/global_step‚899 2loss/SRqqt2/assignment_loss‚1.4894799258811084 /loss/FRt1/detection_loss‚1.3729976577232428 % _timestamp‚1.7844934575232143e+09 0loss/FRt2/assignment_loss‚3.9226988036842907Vìþ /loss/FRt2/detection_loss‚0.5976629577184498 1loss/SRqqt2/detection_loss‚2.3796535588514574 _runtime‚ 614.12475215 _step‚73 1loss/SRqqt1/assignment_loss‚1.138598061792476 0loss/SRqqt1/detection_loss‚0.839197559505646 epoch‚2 /loss/FRt1/assignment_loss‚2.286610598445462 1loss/FBt1/assignment_loss‚0.19199671171608088 1loss/FBt2/assignment_loss‚0.22426784951696954 .loss/FBt1/detection_loss‚2.793809578119745 /loss/FBt2/detection_loss‚3.1479473655227546 &loss/total_loss‚20.384920628477683 trainer/global_step‚899 2loss/SRqqt2/assignment_loss‚1.4894799258811084 /loss/FRt1/detection_loss‚1.3729976577232428 % _timestamp‚1.7844934575232143e+09 0loss/FRt2/assignment_loss‚3.9226988036842907I&u¸5í‚(:ª ’ëôÒñ…¿$ memory_percent‚94.46051120758057 proc.cpu.threads‚17 network.recv‚214392 disk.dm-0.in‚0# disk.nvme0n1.out‚2855.720703125 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 907.58984375 cpu‚25.7814983445544 network.sent‚ 1.215325e+06 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚1500.2294921875 disk.md0.out‚14.6875 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚1500.2294921875* proc.memory.percent‚17.673444747924805) disk./.usagePercent‚76.56143379801551 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚0" proc.memory.rssMB‚ 2895.6171875% disk./.usageGB‚1175.4094123840332 disk.md0.in‚0" disk.dm-0.out‚3522.80224609375! disk.dm-1.out‚801.38916015625 disk.rbd0.out‚0% disk.nvme0c0n1.out‚2855.720703125²“lØœî‚(:‘ ’ëôÒÌö¾¿ gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.802' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.26733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H›mg?ï‚(:´ ¡ëôÒÓ˜¿ disk.nvme1c1n1.in‚0$ memory_percent‚94.45765018463135* proc.memory.percent‚17.673444747924805 cpu‚25.790070338974544% disk.nvme0n1.out‚2861.60498046875 network.recv‚218267 disk.md0.out‚ 14.828125( proc.memory.availableMB‚ 908.05859375 proc.cpu.threads‚17) disk./.usagePercent‚76.56143901596977 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚1500.67626953125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1500.67626953125" proc.memory.rssMB‚ 2895.6171875 network.sent‚ 1.265711e+06% disk./.usageGB‚1175.4094924926758 disk.dm-0.in‚0" disk.dm-0.out‚3528.72412109375! disk.dm-1.out‚801.47509765625' disk.nvme0c0n1.out‚2861.60498046875 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0Vþ‚Nžð‚(:“ ¡ëôÒµåÊ¿ gpu.0.gpu‚91.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚225.826' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.27533333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎPYÿ8ñ‚(:­ °ëôÒ´£…¿ disk.md0.out‚ 15.2109375" disk.dm-0.out‚3529.38818359375 network.recv‚221763% disk./.usageGB‚1175.4095153808594 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1501.33349609375 disk.nvme1n1.in‚0 disk.md0.in‚0% disk.nvme1n1.out‚1501.33349609375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚2862.556640625# memory_percent‚94.4591760635376* proc.memory.percent‚17.673444747924805 cpu‚25.784996714996737 network.sent‚ 1.2688e+06 disk.nvme0n1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚801.5712890625( proc.memory.availableMB‚ 907.80859375" proc.memory.rssMB‚ 2895.6171875) disk./.usagePercent‚76.56144050681384# disk.nvme0n1.out‚2862.556640625 disk.dm-0.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17„U|Ôžò‚(:“ °ëôÒ‹£µ¿ gpu.0.gpu‚97.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚230.152' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.71733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ÂdÌ0ó+ _wandbruntime‚645  _runtime‚645¯W¤™Äô¾ trainer/global_step‚949 $ _timestamp‚1.784493488800375e+09 _runtime‚ 645.40222237 _step‚74 ' lr-AdamW/pg1‚0.00016013983047085398 ' lr-AdamW/pg2‚0.00016013983047085398¡²Èõ ' lr-AdamW/pg1‚0.00016013983047085398 ' lr-AdamW/pg2‚0.00016013983047085398 trainer/global_step‚949 $ _timestamp‚1.784493488800375e+09 _runtime‚ 645.40222237 _step‚74JŸâˆÛ0ö+ _wandbruntime‚645  _runtime‚645DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74183496040960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74183601739776 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7169837 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9536477 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779257922048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚75 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9526335 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7170024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74183601739776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74183496040960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 779169262592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 803184989696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73404326778368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779257922048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777024769433 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54982830100992å.ÈÛø·O _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚73404343817728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5021011 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5021007 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55786109200384 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55785889840640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9526335 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7170024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74183601739776 NFDeviceStatsMonitor.onY8l â"_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74183496040960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 779169262592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 803184989696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73404326778368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779257922048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚949 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777024769433 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54982830100992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7495121 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74183496040960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74183601739776 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7169837 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9536477 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779257922048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚75 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779954978304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74259537128960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7177004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _step‚76 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7502752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2150941 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 803866065408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74259537128960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780043644928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55843057441280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9546311 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72615661380448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 645.806731849 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7177194 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73393556431015 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73479476444672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55842833894400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55038967828992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5026067 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73479493484032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73479493484032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55039061938688 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7502752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7502750 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72615644341088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777895050567 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Þu??Ç&ûÁM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7502750 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7177004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _step‚76 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7502752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2150941 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 803866065408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74259537128960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780043644928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55843057441280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9546311 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72615661380448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 645.806731849 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7177194 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73393556431015 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73479476444672 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55842833894400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55038967828992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5026067 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73479493484032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73479493484032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55039061938688 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7502752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7502750 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72615644341088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777895050567 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9546311 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9536155 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 780043644928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9536155 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 777808264033 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5026063 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 803995502592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17038905 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779954978304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74259537128960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 818519202816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77161827132928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2256557 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10019187 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76255880085296 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7530364 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17895758 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7876573 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚206 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57789810610688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10029329 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5273811 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7876571 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 816350086663 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 676.59135416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77161810093568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17905902 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77980329296384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚77980434995200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77072247211319 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 818519202816 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10029329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7876573 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77980329296384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76255897124656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 818607862272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 818607862272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2256759 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58633336040960 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77072143385537 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 % _timestamp‚1.7844935199884586e+09 _step‚79 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 676.59135416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77161810093568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17905902 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77980329296384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚77980434995200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77072247211319 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 818519202816 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10029329 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7876573 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77980329296384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76255897124656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 818607862272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 818607862272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2256759 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58633336040960 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77072143385537 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 % _timestamp‚1.7844935199884586e+09 _step‚79 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 818519202816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77161827132928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚202 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2256557 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10019187 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76255880085296 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7530364O­ŽVÅ0‰+ _wandbruntime‚676  _runtime‚676½õ'äÇ&ŠÁM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2258861 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 844555696128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77147321231423 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17923365 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77236976799232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5278863 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76330187650048 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 817046794841 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78056370384384 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58690280487936 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78056370384384 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 % _timestamp‚1.7844935203909013e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10029007 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5278867 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58690504034816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7537728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17913207 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57845854228992 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 844426258944 trainer/global_step‚999 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78056264678400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77236959759872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚190 _runtime‚ 676.992840267 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7884200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77236976799232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76330170610688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57845948338688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17913207 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 819393585152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10039163 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77236959759872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 819393585152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10029007 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 819304918528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10039163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7537534 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚80ÛÂBœ‹ÅM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7537534 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚80 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2258861 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 844555696128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77147321231423 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17923365 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77236976799232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5278863 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76330187650048 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 817046794841 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78056370384384 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58690280487936 YIDeviceStatsMâ¬eY/ onitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78056370384384 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 % _timestamp‚1.7844935203909013e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10029007 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5278867 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58690504034816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7537728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17913207 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57845854228992 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 844426258944 trainer/global_step‚999 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78056264678400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77236959759872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚190 _runtime‚ 676.992840267 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7884200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77236976799232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76330170610688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57845948338688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17913207 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 819393585152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10039163 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77236959759872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 819393585152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10029007 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 819304918528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10039163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812P®ºÒ0Œ+ _wandbruntime‚676  _runtime‚676ÿ²××ú 1loss/SRqqt1/detection_loss‚0.7177697715216704 /loss/FBt1/detection_loss‚2.9605238436241534 trainer/global_step‚999 % _timestamp‚1.7844935203918362e+09 _runtime‚ 676.993340198 0loss/FRt1/assignment_loss‚1.8986235375411682 .loss/FRt1/detection_loss‚1.360700002733043 1loss/SRqqt2/detection_loss‚2.8222991467703187 .loss/FBt2/detection_loss‚2.596181161213335 /loss/FRt2/assignment_loss‚3.746808182495544 2loss/SRqqt1/assignment_loss‚1.1815107047598217 1loss/FBt2/assignment_loss‚0.19883512589240127 .loss/FRt2/detection_loss‚0.583578719323902 &loss/total_loss‚19.570572902527374 epoch‚2 _step‚81 1loss/SRqqt2/assignment_loss‚1.424978871490228 1loss/FBt1/assignment_loss‚0.07876383516178753žE'›Žþ epoch‚2 _step‚81 1loss/SRqqt2/assignment_loss‚1.424978871490228 1loss/FBt1/assignment_loss‚0.07876383516178753 1loss/SRqqt1/detection_loss‚0.7177697715216704 /loss/FBt1/detection_loss‚2.9605238436241534 trainer/global_step‚999 % _timestamp‚1.7844935203918362e+09 _runtime‚ 676.993340198 0loss/FRt1/assignment_loss‚1.8986235375411682 .loss/FRt1/detection_loss‚1.360700002733043 1loss/SRqqt2/detection_loss‚2.8222991467703187 .loss/FBt2/detection_loss‚2.596181161213335 /loss/FRt2/assignment_loss‚3.746808182495544 2loss/SRqqt1/assignment_loss‚1.1815107047598217 1loss/FBt2/assignment_loss‚0.19883512589240127 .loss/FRt2/detection_loss‚0.583578719323902 &loss/total_loss‚19.570572902527374QÑQßA‚(:¶ Ýëô҃昿 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme0c0n1.out‚2933.03369140625' disk.nvme1c1n1.out‚1503.63232421875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1503.63232421875 disk.md0.in‚0 disk.md0.out‚ 16.23828125" disk.dm-0.out‚3599.49755859375 disk.dm-1.in‚0% disk.nvme0n1.out‚2933.03369140625 disk.dm-0.in‚0* proc.memory.percent‚17.673540115356445 cpu‚25.778485691740258 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0! disk.dm-1.out‚802.01025390625$ memory_percent‚94.85599994659424( proc.memory.availableMB‚ 842.79296875" proc.memory.rssMB‚ 2895.6328125 network.sent‚ 1.353697e+06 network.recv‚233635) disk./.usagePercent‚76.54985217585313% disk./.usageGB‚1175.2316055297852 disk.rbd0.in‚0 disk.rbd0.out‚0›62æž‚(:“ ÝëôÒÄ÷Ï¿ gpu.0.gpu‚55.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚117.817' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.27233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IàT:‘‚(:¯ ìëôÒøà–Á proc.cpu.threads‚17 network.sent‚ 1.404785e+06 network.recv‚237756% disk./.usageGB‚1175.2316627502441 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1504.24951171875! disk.dm-1.out‚802.12451171875 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.md0.out‚16.625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚2938.16943359375% disk.nvme0n1.out‚2938.16943359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1504.24951171875$ memory_percent‚94.85630989074707' proc.memory.availableMB‚ 842.7421875( disk./.usagePercent‚76.5498559029633 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 2895.63671875* proc.memory.percent‚17.673563957214355 cpu‚25.78817470889637 disk.md0.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚3604.30224609375Åc­~Ÿ’‚(:” ìëôÒÝÍÏÁ gpu.0.gpu‚90.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚154.322' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚51.440666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëS[0“+ _wandbruntime‚707  _runtime‚707è—åQÅ”¿ & lr-AdamW/pg2‚0.0001600165618000322 trainer/global_step‚1049 % _timestamp‚1.7844935511940007e+09 _runtime‚ 707.795757869 _step‚82 & lr-AdamW/pg1‚0.0001600165618000322*€A™É•Ã & lr-AdamW/pg1‚0.0001600165618000322 & lr-AdamW/pg2‚0.0001600165618000322 trainer/global_step‚1049 % _timestamp‚1.7844935511940007e+09 _runtime‚ 707.795757869 _step‚82R[/¿L0–+ _wandbruntime‚707  _runtime‚707}¼²½'—·O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5526611 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213068288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81777268250624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10512039 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2364572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80919293408768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79970406354896 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8258023 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258021 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81777162551808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80826012011727 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 857957802496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7891183 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5526607 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79970423394256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 857869143040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61480762973184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 _runtime‚ 707.79653584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80919310448128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60596697010688 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60596602900992 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 855588617471 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10522181 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 857957802496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 857869143040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7891008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 884279030784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144"‰ÚØ%˜»O YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61480762973184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 _runtime‚ 707.79653584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80919310448128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60596697010688 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60596602900992 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 855588617471 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10522181 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 857957802496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 857869143040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7891008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 884279030784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5526611 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213068288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81777268250624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10512039 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2364572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80919293408768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79970406354896 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8258023 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258021 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81777162551808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80826012011727 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 857957802496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7891183 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5526607 _LDeviceStatsMonit…z³3éor.on_train_batch_start/requested_bytes.large_pool.allocated‚79970423394256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 857869143040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509SÔÅ«§0™+ _wandbruntime‚708  _runtime‚708‚uåÆ&šÀM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2366720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 884964431872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8265652 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81853203639808 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 858743525376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18787509 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61537924413440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80900982205937 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _runtime‚ 708.21463892 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81853203639808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2366535 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125242880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 856372112183 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8265650 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80044696880288 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8265652 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80994443075072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80994460114432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80994443075072 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7898387 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81853097933824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7898198 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5531667 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18797667 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80994460114432 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 858743525376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10521859 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 885089674752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130׌°6Ê&›ÄM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81853203639808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2366535 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125242880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 856372112183 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8265650 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80044696880288 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8265652 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80994443075072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80994460114432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80994443075072 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7898387 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81853097933824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7898198 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5531667 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18797667 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80994460114432 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 858743525376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10521859 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 885089674752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2366720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 884964431872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8265652 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81853203639808 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 858743525376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18787509 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61537924413440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80900982205937 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _runtime‚ 708.21463892TdåG°0œ+ _wandbruntime‚708  _runtime‚708ªu ?ü /loss/FRt2/assignment_loss‚3.739678327735388 2loss/SRqqt1/assignment_loss‚1.1065972373316675 2loss/SRqqt2/assignment_loss‚1.4187636563487083 epoch‚2 % _timestamp‚1.7844935516137257e+09 /loss/FRt1/assignment_loss‚1.946709997015085 /loss/FRt1/detection_loss‚1.3626583752987511 1loss/SRqqt1/detection_loss‚0.7475732863965296 /loss/FBt1/detection_loss‚1.7722545393950646 _step‚85 1loss/FBt1/assignment_loss‚0.12415084745869752 1loss/SRqqt2/detection_loss‚2.3723374060908355 %loss/total_loss‚19.46631368678414 trainer/global_step‚1049 1loss/FBt2/assignment_loss‚0.15747287998766085 /loss/FRt2/detection_loss‚0.5253467383056437 .loss/FBt2/detection_loss‚4.192770395420107 _runtime‚ 708.215252681ŒÄ枀 1loss/SRqqt2/detection_loss‚2.3723374060908355 %loss/total_loss‚19.46631368678414 trainer/global_step‚1049 1loss/FBt2/assignment_loss‚0.15747287998766085 /loss/FRt2/detection_loss‚0.5253467383056437 .loss/FBt2/detection_loss‚4.192770395420107 _runtime‚ 708.215252681 /loss/FRt2/assignment_loss‚3.739678327735388 2loss/SRqqt1/assignment_loss‚1.1065972373316675 2loss/SRqqt2/assignment_loss‚1.4187636563487083 epoch‚2 % _timestamp‚1.7844935516137257e+09 /loss/FRt1/assignment_loss‚1.946709997015085 /loss/FRt1/detection_loss‚1.3626583752987511 1loss/SRqqt1/detection_loss‚0.7475732863965296 /loss/FBt1/detection_loss‚1.7722545393950646 _step‚85 1loss/FBt1/assignment_loss‚0.12415084745869752UÂ5Ë2Ÿ‚(:§ ûëôÒÙùì¾' disk.nvme1c1n1.out‚1506.62255859375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1506.62255859375 disk.md0.out‚ 16.93359375 disk.nvme0c0n1.in‚0 disk.md0.in‚0 disk.dm-0.out‚3616.802734375 disk.nvme0n1.out‚ 2949.046875# memory_percent‚94.8568344116211 disk.dm-0.in‚0 network.sent‚ 1.407749e+06 network.recv‚241330) disk./.usagePercent‚76.54982086812765 disk.rbd0.in‚0# proc.memory.rssMB‚ 2895.63671875* proc.memory.percent‚17.673563957214355 disk.nvme1c1n1.in‚0! disk.dm-1.out‚802.20654296875 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 2949.046875 disk.nvme0n1.in‚0% proc.memory.availableMB‚ 842.65625 cpu‚25.795741253643307 proc.cpu.threads‚17 disk.dm-1.in‚0% disk./.usageGB‚1175.2311248779297©Íûyž ‚(:“ ûëôÒËïÀ¿ gpu.0.gpu‚90.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚184.518' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.50600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sWñ\9¡‚(:® ŠìôÒƒžò¾ disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚2949.89892578125 disk.nvme0n1.in‚0 disk.dm-0.out‚3617.369140625 disk.rbd0.out‚0( proc.memory.availableMB‚ 842.65234375 proc.cpu.threads‚17 network.sent‚ 1.47102e+06% disk./.usageGB‚1175.2311553955078% disk.nvme0n1.out‚2949.89892578125' disk.nvme1c1n1.out‚1507.31298828125% disk.nvme1n1.out‚1507.31298828125 disk.dm-0.in‚0" memory_percent‚94.856858253479 network.recv‚245657) disk./.usagePercent‚76.54982285591973 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.out‚ 802.416015625 disk.rbd0.in‚0* proc.memory.percent‚17.673563957214355 disk.nvme1c1n1.in‚0 disk.md0.out‚ 17.2890625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2895.63671875 cpu‚25.80884268499811±Xž¢‚(:“ ŠìôÒÝ‚²¿ gpu.0.gpu‚97.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚227.821' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.94033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LÑ|Ï0£+ _wandbruntime‚739  _runtime‚739زҼŤ¿ trainer/global_step‚1099 % _timestamp‚1.7844935827558432e+09 _runtime‚ 739.357663515 _step‚86 & lr-AdamW/pg1‚0.0001599472381200386 & lr-AdamW/pg2‚0.0001599472381200386È•¹É¥Ã & lr-AdamW/pg1‚0.0001599472381200386 & lr-AdamW/pg2‚0.0001599472381200386 trainer/global_step‚1099 % _timestamp‚1.7844935827558432e+09 _runtime‚ 739.357663515 _step‚86VüñØ0¦+ _wandbruntime‚739  _runtime‚739¹4§¸O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85574101506048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19644362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64328117021184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚ 739.358471877 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84676793763328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63403583410688 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85573995807232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 924755067392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7844935827566924e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5779407 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8250737 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 894827148279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64328338478080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8639473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11004891 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚1099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63403489300992 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8639473 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84676793763328 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85574101506048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897219083264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579672986353 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11015033 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5779411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2471511 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 894740361857 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 924627720192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11015033 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _step‚87 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 897307742720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11004891 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8639471 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 897307742720 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8250922 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2471330 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segmeNßc¤ nt.small_pool.allocated‚130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84676776723968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897219083264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85573995807232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83684932624496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19654506 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83684949663856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84676776723968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760·‹Â'¨¼O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8639473 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84676793763328 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85574101506048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897219083264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579672986353 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11015033 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5779411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2471511 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 894740361857 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 924627720192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11015033 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _step‚87 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 897307742720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11004891 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8639471 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 897307742720 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8250922 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2471330 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84676776723968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 897219083264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85573995807232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127347200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83684932624496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19654506 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83684949663856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84676776723968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85574101506048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19644362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64328117021184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚ 739.358471877 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84676793763328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63403583410688 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85573995807232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 924755067392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7844935827566924e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5779407 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8250737 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 894827148279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚185 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64328338478080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8639473 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11004891 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚1099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63403489300992W,¤à0©+ _wandbruntime‚739  _runtime‚739pZ9ûÈ&ªÂM ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85650036895232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 925431293440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647100 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8257810 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2473514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84751943429632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5784463 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 898093465600 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898004798976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8647102 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85650036895232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚167 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83759240189248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11024867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227741184 _runtime‚ 739.752324377 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19671969 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19661811 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84654747006345 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11024867 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 895523856457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647100 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11014711 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 % _timestamp‚1.7844935831503081e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5784467 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83759223149888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1099 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 898093465600 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64385286063616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 925564924928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 % _timestamp‚1.7844935831503081e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5784467 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83759223149888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1099 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 898093465600 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64385286063616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 925564924928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85650036895232 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 925431293440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647100 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8257810 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2473514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84751943429632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5784463 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 898093465600 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898004798976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8647102 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85650036895232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚167 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83759240189248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11024867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227741184Xœ°H÷0¬+ _wandbruntime‚739  _runtime‚739rEÌu­þ .loss/FRt2/detection_loss‚0.580355045734396 .loss/FBt1/detection_loss‚2.533865178975466 epoch‚2 s^È×trainer/global_step‚1099 _runtime‚ 739.753331579 _step‚89 0loss/FRt1/assignment_loss‚1.8340987168264165 2loss/SRqqt1/assignment_loss‚1.1466234425736064 1loss/FBt2/assignment_loss‚0.15402666772265372 /loss/FRt1/detection_loss‚1.4318610166263104 1loss/SRqqt2/detection_loss‚1.7089490591670222 /loss/FBt2/detection_loss‚2.3572660366166343 &loss/total_loss‚17.539482447362722 0loss/FRt2/assignment_loss‚3.5047215348325866 1loss/SRqqt1/detection_loss‚0.7542402766386523 % _timestamp‚1.7844935831517925e+09 2loss/SRqqt2/assignment_loss‚1.3456466894022667 1loss/FBt1/assignment_loss‚0.18782878224670907Û7ns®‚ 2loss/SRqqt2/assignment_loss‚1.3456466894022667 1loss/FBt1/assignment_loss‚0.18782878224670907 .loss/FRt2/detection_loss‚0.580355045734396 .loss/FBt1/detection_loss‚2.533865178975466 epoch‚2 trainer/global_step‚1099 _runtime‚ 739.753331579 _step‚89 0loss/FRt1/assignment_loss‚1.8340987168264165 2loss/SRqqt1/assignment_loss‚1.1466234425736064 1loss/FBt2/assignment_loss‚0.15402666772265372 /loss/FRt1/detection_loss‚1.4318610166263104 1loss/SRqqt2/detection_loss‚1.7089490591670222 /loss/FBt2/detection_loss‚2.3572660366166343 &loss/total_loss‚17.539482447362722 0loss/FRt2/assignment_loss‚3.5047215348325866 1loss/SRqqt1/detection_loss‚0.7542402766386523 % _timestamp‚1.7844935831517925e+09Yv-Æ05¯‚(:ª ™ìôÒ캿$ disk.nvme1n1.out‚1507.9677734375* proc.memory.percent‚17.673563957214355 disk.nvme1n1.in‚0 network.sent‚ 1.487585e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0# disk.nvme0n1.out‚2963.650390625 disk.md0.out‚ 17.51171875! disk.dm-0.out‚3631.1611328125 disk.dm-1.in‚0) disk./.usagePercent‚76.56851630124687 disk.rbd0.out‚0 disk.md0.in‚0 disk.dm-0.in‚0$ memory_percent‚94.85893249511719$ proc.memory.availableMB‚842.3125# proc.memory.rssMB‚ 2895.63671875 cpu‚25.820410461141158% disk./.usageGB‚1175.5181465148926% disk.nvme0c0n1.out‚2963.650390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1507.9677734375 disk.dm-1.out‚ 802.533203125 proc.cpu.threads‚17 network.recv‚249265 disk.nvme0n1.in‚0˜+otœ°‚(:‘ ™ìôÒ€…­¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.831' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.61033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µyê‰5±‚(:ª ¨ìôÒ™œå¾# disk.nvme0n1.out‚3132.212890625& disk.nvme1c1n1.out‚1575.4384765625! disk.dm-0.out‚3866.0048828125% disk.nvme0c0n1.out‚3132.212890625% disk./.usageGB‚1175.4689445495605 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚1575.4384765625 disk.md0.in‚0 disk.md0.out‚ 17.9765625 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.538719e+06 network.recv‚253421 disk.dm-1.out‚ 802.73046875 disk.nvme0c0n1.in‚0$ memory_percent‚94.85905170440674* proc.memory.percent‚17.673540115356445 cpu‚25.82680424720317 proc.cpu.threads‚17 disk.nvme1n1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 842.29296875" proc.memory.rssMB‚ 2895.6328125) disk./.usagePercent‚76.56531148344312 disk.nvme0n1.in‚0 ^kž²‚(:“ ¨ìôÒ©Ó×¾ gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚232.928' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.64266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}sGc0³+ _wandbruntime‚770  _runtime‚770¶JüÅ´¿ trainer/global_step‚1149 % _timestamp‚1.7844936141767023e+09 _runtime‚ 770.778530977 _step‚90 & lr-AdamW/pg1‚0.0001598727940373105 & lr-AdamW/pg2‚0.0001598727940373105dLɵà _step‚90 & lr-AdamW/pg1‚0.0001598727940373105 & lr-AdamW/pg2‚0.0001598727940373105 trainer/global_step‚1149 % _timestamp‚1.7844936141767023e+09 _runtime‚ 770.778530977ZÍS«t0¶+ _wandbruntime‚770  _runtime‚770J°=½'··O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 _step‚91 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9020923 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67175438873600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 933978892665 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11497743 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11497743 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 936657682944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89370829062656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 965063172608 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88333541612543 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 934065679087 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8609568 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8609381 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89370829062656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 936657682944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89370934761472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88434260039168 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89370934761472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67175664524800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88333437786761 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _runtime‚ 770.779333949 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11497743 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11497743 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 936657682944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89370829062656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 965063172608 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88333541612543 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 934065679087 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8609568 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8609381 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89370829062656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 936657682944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89370934761472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88434260039168 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89370934761472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67175664524800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88333437786761 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _runtime‚ 770.779333949 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 771.182481325 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 937354739200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8616758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2579491 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11507563 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1149 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87473766458848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9028550 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2579318 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66266607538688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20536113 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 934849173799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8616581 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88408511806753 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 937443405824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9028552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88509426744832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 934762387265 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89446870150656 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚173 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6037267 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚92 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88509409705472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584u#îƒÌ&»ÆM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚173 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6037267 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚92 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88509409705472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 771.182481325 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 937354739200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8616758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2579491 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11507563 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚1149 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87473766458848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9028550 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2579318 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66266607538688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20536113 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 934849173799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8616581 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88408511806753 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 937443405824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129437184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786534 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825894 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9028552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88509426744832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 934762387265 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89446870150656 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0\c¢–û0¼+ _wandbruntime‚771  _runtime‚771³$S½ü 1loss/SRqqt1/detection_loss‚0.7373105389688313 .loss/FBt2/detection_loss‚4.251683838838083 epoch‚2 0loss/FRt1/assignment_loss‚1.7825209826504418 1loss/FBt1/assignment_loss‚0.15851362983888417 /loss/FRt2/detection_loss‚0.6130230604319465 0loss/SRqqt2/detection_loss‚2.104148810848668 _step‚93 /loss/FBt1/detection_loss‚3.1145314047845054 trainer/global_step‚1149 % _timestamp‚1.7844936145814114e+09 /loss/FRt2/assignment_loss‚3.376450184204569 2loss/SRqqt1/assignment_loss‚1.0267129481752189 2loss/SRqqt2/assignment_loss‚1.3237100460037596 1loss/FBt2/assignment_loss‚0.24389102311637836 /loss/FRt1/detection_loss‚1.3837608610983334 %loss/total_loss‚20.11625732895962 _runtime‚ 771.183040856Ö h¾€ /loss/FRt2/assignment_loss‚3.376450184204569 2loss/SRqqt1/assignment_loss‚1.0267129481752189 2loss/SRqqt2/assignment_loss‚1.3237100460037596 1loss/FBt2/assignment_loss‚0.24389102311637836 /loss/FRt1/detection_loss‚1.3837608610983334 %loss/total_loss‚20.11625732895962 _runtime‚ 771.183040856 1loss/SRqqt1/detection_loss‚0.7373105389688313 .loss/FBt2/detection_loss‚4.251683838838083 epoch‚2 0loss/FRt1/assignment_loss‚1.7825209826504418 1loss/FBt1/assignment_loss‚0.15851362983888417 /loss/FRt2/detection_loss‚0.6130230604319465 0loss/SRqqt2/detection_loss‚2.104148810848668 _step‚93 /loss/FBt1/detection_loss‚3.1145314047845054 trainer/global_step‚1149 % _timestamp‚1.7844936145814114e+09]¦Á\Å2¿‚(:§ ·ìôÒ‹ŽÛ¾ network.recv‚257050) disk./.usagePercent‚76.38833934719953% disk./.usageGB‚1172.7519798278809 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚1578.3701171875 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 802.80859375 disk.rbd0.in‚0% disk.nvme0c0n1.out‚3148.892578125& disk.nvme1c1n1.out‚1578.3701171875$ memory_percent‚94.85945701599121' proc.memory.availableMB‚ 842.2265625* proc.memory.percent‚17.673540115356445 network.sent‚ 1.541683e+06 disk.dm-0.in‚0! disk.dm-0.out‚3884.5576171875 disk.rbd0.out‚0 disk.md0.out‚18.4375" proc.memory.rssMB‚ 2895.6328125 proc.cpu.threads‚17 disk.nvme0n1.in‚0# disk.nvme0n1.out‚3148.892578125 cpu‚25.8332309916898569ùJžÀ‚(:“ ·ìôÒ¥Âɾ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚225.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.21266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/RçÖ7Á‚(:¬ ÆìôÒ¼‘í¾ disk.dm-0.in‚0! disk.dm-0.out‚3885.2685546875& disk.nvme1c1n1.out‚1579.1650390625# proc.memory.rssMB‚ 2895.63671875* proc.memory.percent‚17.673563957214355 network.sent‚ 1.603708e+06) disk./.usagePercent‚76.21133267306831% disk./.usageGB‚1170.0344848632812% disk.nvme0c0n1.out‚3150.041015625 cpu‚25.829205725996673 network.recv‚261427$ disk.nvme1n1.out‚1579.1650390625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0# disk.nvme0n1.out‚3150.041015625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 18.95703125 disk.dm-1.out‚ 802.916015625 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ memory_percent‚94.86055374145508& proc.memory.availableMB‚ 842.046875 proc.cpu.threads‚17àôŽÞžÂ‚(:“ ÆìôÒâ³»¿ gpu.0.gpu‚88.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚231.973' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.32433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ³I÷0Ã+ _wandbruntime‚802  _runtime‚802ÎþrfÇÄÁ ' lr-AdamW/pg1‚0.00015979323433325216 ' lr-AdamW/pg2‚0.00015979323433325216 trainer/global_step‚1199 % _timestamp‚1.7844936455539362e+09 _runtime‚ 802.157275979 _step‚94eÂ`ËÅÅ % _timestamp‚1.7844936455539362e+09 _runtime‚ 802.157275979 _step‚94 ' lr-AdamW/pg1‚0.00015979323433325216 ' lr-AdamW/pg2‚0.00015979323433325216 trainer/global_step‚1199^<“¥à0Æ+ _wandbruntime‚802  _runtime‚802ëpÀ'ǺO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6285011 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69017262100992 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21403110 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9402371 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 976007623168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12000737 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91113985163696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105698816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21392966 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚177 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93167662318080 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92087306412951 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92191760393728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 973217423473 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70022793478144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚95 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9402373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1005531377152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2683064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69017356210688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125250048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1199 _runtime‚ 802.157377519 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11990595 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 % _timestamp‚1.7844936455545661e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 975918963712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91114002203056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105698816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93167662318080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92191743354368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93167768016896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8968075 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 976007623168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 973304209895 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70023012837888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11990595 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93167768016896 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9402373 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 975918963712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2682887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92191743354368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12000737 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92191760393728 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8967894 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93167662318080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92191743354368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301876736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417275 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93167768016896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8968075 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659456 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 976007623168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 973304209895 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70023012837888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301876736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11990595 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93167768016896 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9402373 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 975918963712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2682887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92191743354368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12000737 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92191760393728 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8967894 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6285011 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69017262100992 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21403110 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9402371 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 976007623168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786422 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12000737 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91113985163696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_stý˜\‰art/allocated_bytes.all.current‚ 105698816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21392966 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚177 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93167662318080 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92087306412951 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92191760393728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 973217423473 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70022793478144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚95 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9402373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1005531377152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2683064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69017356210688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125250048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1199 _runtime‚ 802.157377519 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11990595 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825782 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 % _timestamp‚1.7844936455545661e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 975918963712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91114002203056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0_ìò‘Ø0É+ _wandbruntime‚802  _runtime‚802`¹ Ê&ÊÄM QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9410002 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69073493938688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1006470261248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92266893020672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231935488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69073399828992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21410415 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92266910060032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9410000 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 976793346048 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 976704679424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9410002 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 802.556577233 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21410415 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93243703406080 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 976704679424 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9410000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12010571 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91188292728448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚188 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12010571 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 % _timestamp‚1.7844936459545627e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8975041 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70079732264448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91188275689088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12000415 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92266893020672 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8975229 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0<úOÎ&ËÈM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69073493938688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1006470261248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417275 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92266893020672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105705984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231935488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69073399828992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88666624 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301876736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21410415 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92266910060032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9410000 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 976793346048 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 976704679424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9410002 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301876736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 802.556577233 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21410415 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93243703406080 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 976704679424 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9410000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12010571 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91188292728448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚188 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12010571 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105705984 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 % _timestamp‚1.7844936459545627e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8975041 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70079732264448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91188275689088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12000415 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92266893020672 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8975229 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9410002`\Ò}Ï0Ì+ _wandbruntime‚802  _runtime‚802Va™,Íü /loss/FRt2/assignment_loss‚3.409344079969554 2loss/SRqqt2/assignment_loss‚1.2373281987894276 0loss/FBt1/assignment_loss‚0.2072696773166924 /loss/FRt1/detection_loss‚1.3220774431395266 epoch‚2 % _timestamp‚1.7844936459555147e+09 2loss/SRqqt1/assignment_loss‚1.1073199300899763 /loss/FRt2/detection_loss‚0.5938252538884646 1loss/SRqqt1/detection_loss‚0.6892655732981551 &loss/total_loss‚17.948794718754172 0loss/SRqqt2/detection_loss‚2.078150334830179 /loss/FBt1/detection_loss‚3.0970882540902482 /loss/FBt2/detection_loss‚2.0547994102787985 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚1.8986522334490636 0loss/FBt2/assignment_loss‚0.2536743296140834 _runtime‚ 802.557072394 _step‚976-¤Î€ /loss/FRt2/assignment_loss‚3.409344079969554 2loss/SRqqt2/assignment_loss‚1.2373281987894276 0loss/FBt1/assignment_loss‚0.2072696773166924 /loss/FRt1/detection_loss‚1.3220774431395266 epoch‚2 % _timestamp‚1.7844936459555147e+09 2loss/SRqqt1/assignment_loss‚1.1073199300899763 /loss/FRt2/detection_loss‚0.5938252538884646 1loss/SRqqt1/detection_loss‚0.6892655732981551 &loss/total_loss‚17.948794718754172 0loss/SRqqt2/detection_loss‚2.078150334830179 /loss/FBt1/detection_loss‚3.0970882540902482 /loss/FBt2/detection_loss‚2.0547994102787985 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚1.8986522334490636 0loss/FBt2/assignment_loss‚0.2536743296140834 _runtime‚ 802.557072394 _step‚97afþHÔ9Ï‚(:® ÕìôÒ¬ý–¿ proc.cpu.threads‚17 network.recv‚264956 disk.nvme1n1.in‚0 disk.md0.out‚ 19.12890625" disk.dm-0.out‚3889.90087890625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚94.93348598480225! proc.memory.rssMB‚ 2907.578125 network.sent‚ 1.622394e+06 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1579.7197265625$ disk.nvme1n1.out‚1579.7197265625 disk.rbd0.out‚0% disk.nvme0n1.out‚3154.60693359375 cpu‚25.85383757617894$ disk./.usageGB‚1170.034610748291 disk.dm-1.out‚ 803.013671875' disk.nvme0c0n1.out‚3154.60693359375( proc.memory.availableMB‚ 830.09765625( disk./.usagePercent‚76.2113408727107 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0* proc.memory.percent‚17.746448516845703w E|ŸÐ‚(:” ÕìôÒóëÉ¿ gpu.0.gpu‚98.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚127.805' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚42.601666666666674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼¸¬0Ñ+ _wandbruntime‚815  _runtime‚815EæÏÙ"Òü{ 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13272105157375336 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1330246925354004 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5157705545425415 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.514525294303894 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.328125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5259577631950378 trainer/global_step‚1199 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9593147039413452 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06578284502029419 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.47747543454170227 6validation_average_jet_accuracy‚0.2909258306026459 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1771998107433319 *jet/accuracy_1_of_3‚0.8155583143234253 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.30245986580848694 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.799271821975708 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.14844919741153717 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3756730556488037 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5658029913902283 particle/accuracy_3_of_4‚1 $ _timestamp‚1.784493658909593e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.30040210485458374 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,jet/accuracy_4_of_4‚0.002801120514050126 particle/accuracy_2_of_3‚1 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.692307710647583 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8066763281822205 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12815767526626587 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 *jet/accuracy_2_of_3‚0.3294459879398346 +jet/accuracy_2_of_4‚0.32542043924331665 ,jet/accuracy_3_of_4‚0.027321230620145798 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.514525294303894 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.992007315158844 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9938672184944153 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5777983665466309 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5199877619743347 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5453589558601379 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9175810813903809 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5105452537536621 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6508314609527588 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.15260478854179382 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6782501935958862 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6580513119697571 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9985868334770203 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2168412208557129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.14403891563415527 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 _step‚98 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5373377799987793 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9787282347679138 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16204825043678284 +jet/accuracy_2_of_2‚0.09034042060375214 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18688495457172394 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9879732728004456 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48289334774017334 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.580488383769989 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 &particle/accuracy‚0.81928551197052 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9922696352005005 particle/accuracy_2_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0056238300167024136 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11467857658863068 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9713862538337708 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03102831356227398 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚2 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5424582958221436 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/eveÕzà'nt_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.25865837931632996 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44921278953552246 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004575259517878294 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12092044204473495 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9248209595680237 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12139450013637543 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024419804103672504 *jet/accuracy_1_of_4‚0.8161526918411255 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18508188426494598 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35624998807907104 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_5_of_5‚0 *loss/val_total_loss‚18.431781833956062 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09980065375566483 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09392949938774109 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19091832637786865 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7001972198486328 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9838097095489502 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9909931421279907 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.941798985004425 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8518518209457397 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48859673738479614 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06833334267139435 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.0625 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5740740895271301 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7487401962280273 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5903538465499878 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4637361168861389 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5053744316101074 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6352383494377136 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚ 815.511844312 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1646851897239685 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3715340793132782 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9704135656356812 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12139450013637543 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21338362991809845 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.33156755566596985 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23844000697135925 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2596439719200134 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3436093330383301 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02601170539855957 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.38408035039901733 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_3_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4932861924171448 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8518518209457397 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.32108789682388306 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9821932911872864 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22608627378940582 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.30512183904647827 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.0714285746216774 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8261488080024719 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6005805134773254 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3703376352787018 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06698259711265564 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6120185256004333 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1511755734682083 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.32108789682388306 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.35990262031555176 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12813396751880646 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9898090362548828 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2916666567325592 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5879014134407043 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.46875 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.649470865726471 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.938758134841919 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5497170686721802 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9681878089904785 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9746438264846802 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.24971449375152588 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9664431810379028 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.25865837931632996 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9524859189987183 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3738945722579956 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8496421575546265 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.27173060178756714 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.43757662177085876 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6355153322219849 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3831905722618103 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6032378077507019 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.539700984954834 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5064324140548706 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9749334454536438 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3888888955116272 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.987467348575592 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6944444179534912 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.40542733669281006 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4772000014781952 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9051553010940552 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010889282450079918 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.692307710647583 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1979166716337204 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6427170634269714 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.225866436958313 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.44198790192604065 /particle/accuracy_4_of_4‚0.9645848274230957 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.5769230723381042 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.28125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15410730242729187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22320926189422607 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42748069763183594 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7045105695724487 +jet/accuracy_3_of_3‚0.03758550435304642 'particle/f_score‚0.5880388021469116 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2939814627170563 * Purity/*FRt1SRqqt2FBt/FBt_purity‚0.625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9933419227600098 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3888888955116272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 particle/accuracy_1_of_2‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.3293096125125885 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2416994273662567 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11636194586753845 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.678930401802063 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5740740895271301 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9909931421279907 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.30377981066703796 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9645516276359558 +particle/specificity‚0.8790981769561768 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1|ûô>Ó€| =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5105452537536621 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6508314609527588 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.15260478854179382 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6782501935958862 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6580513119697571 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9985868334770203 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2168412208557129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.14403891563415527 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 _step‚98 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5373377799987793 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9787282347679138 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16204825043678284 +jet/accuracy_2_of_2‚0.09034042060375214 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18688495457172394 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9879732728004456 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48289334774017334 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.580488383769989 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 &particle/accuracy‚0.81928551197052 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9922696352005005 particle/accuracy_2_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0056238300167024136 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11467857658863068 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9713862538337708 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03102831356227398 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚2 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5424582958221436 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.25865837931632996 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44921278953552246 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004575259517878294 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12092044204473495 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9248209595680237 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12139450013637543 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024419804103672504 *jet/accuracy_1_of_4‚0.8161526918411255 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18508188426494598 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35624998807907104 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_5_of_5‚0 *loss/val_total_loss‚18.431781833956062 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09980065375566483 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09392949938774109 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19091832637786865 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7001972198486328 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9838097095489502 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9909931421279907 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.941798985004425 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8518518209457397 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48859673738479614 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06833334267139435 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.0625 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5740740895271301 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7487401962280273 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5903538465499878 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4637361168861389 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5053744316101074 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6352383494377136 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚ 815.511844312 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1646851897239685 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3715340793132782 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9704135656356812 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12139450013637543 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21338362991809845 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.33156755566596985 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23844000697135925 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2596439719200134 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3436093330383301 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02601170539855957 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.38408035039901733 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_3_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4932861924171448 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8518518209457397 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.32108789682388306 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9821932911872864 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22608627378940582 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.30512183904647827 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.0714285746216774 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8261488080024719 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6005805134773254 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3703376352787018 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06698259711265564 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6120185256004333 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1511755734682083 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.32108789682388306 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.35990262031555176 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12813396751880646 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9898090362548828 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2916666567325592 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5879014134407043 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.46875 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.649470865726471 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.938758134841919 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5497170686721802 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9681878089904785 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9746438264846802 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.24971449375152588 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9664431810379028 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.25865837931632996 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9524859189987183 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3738945722579956 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8496421575546265 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.27173060178756714 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.43757662177085876 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6355153322219849 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3831905722618103 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6032378077507019 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.539700984954834 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5064324140548706 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9749334454536438 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3888888955116272 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.987467348575592 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6944444179534912 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.40542733669281006 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4772000014781952 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9051553010940552 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010889282450079918 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.692307710647583 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1979166716337204 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6427170634269714 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.225866436958313 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.44198790192604065 /particle/accuracy_4_of_4‚0.9645848274230957 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.5769230723381042 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.28125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15410730242729187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22320926189422607 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42748069763183594 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7045105695724487 +jet/accuracy_3_of_3‚0.03758550435304642 'particle/f_score‚0.5880388021469116 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2939814627170563 * Purity/*FRt1SRqqt2FBt/FBt_purity‚0.625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9933419227600098 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3888888955116272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 particle/accuracy_1_of_2‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.3293096125125885 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2416994273662567 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11636194586753845 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.678930401802063 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5740740895271301 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9909931421279907 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.30377981066703796 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9645516276359558 +particle/specificity‚0.8790981769561768 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13272105157375336 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1330246925354004 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5157705545425415 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.514525294303894 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.328125 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5259577631950378 trainer/global_step‚1199 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9593147039413452 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06578284502029419 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.47747543454170227 6validation_average_jet_accuracy‚0.2909258306026459 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1771998107433319 *jet/accuracy_1_of_3‚0.8155583143234253 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.30245986580848694 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.799271821975708 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.14844919741153717 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3756730556488037 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5658029913902283 particle/accuracy_3_of_4‚1 $ _timestamp‚1.784493658909593e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.30040210485458374 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,jet/accuracy_4_of_4‚0.002801120514050126 particle/accuracy_2_of_3‚1 8"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.692307710647583 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8066763281822205 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12815767526626587 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 *jet/accuracy_2_of_3‚0.3294459879398346 +jet/accuracy_2_of_4‚0.32542043924331665 ,jet/accuracy_3_of_4‚0.027321230620145798 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.514525294303894 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.992007315158844 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9938672184944153 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5777983665466309 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5199877619743347 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5453589558601379 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9175810813903809 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968bÊñ+ˆ@Ô‚(:µ äìôÒÍí¾ network.sent‚ 1.670894e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚3155.56298828125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1644.29345703125 disk.dm-0.in‚0 network.recv‚269382! disk.dm-1.out‚866.96728515625 disk.rbd0.in‚0 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚1644.29345703125 disk.md0.out‚ 19.56640625$ memory_percent‚95.38741111755371# proc.memory.rssMB‚ 2918.08203125) disk./.usagePercent‚76.21134410287286' proc.memory.availableMB‚ 755.7265625* proc.memory.percent‚17.810559272766113 cpu‚25.86958440030986 proc.cpu.threads‚17% disk./.usageGB‚1170.0346603393555 disk.dm-1.in‚0% disk.nvme0n1.out‚3155.56298828125 disk.md0.in‚0" disk.dm-0.out‚3890.53369140625Ûz›œÕ‚(:‘ äìôÒö¿Ð¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.282' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.42733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®­eAÖ‚(:¶ óìôÒð¸Á¿ disk.dm-0.in‚0" disk.dm-0.out‚3901.36865234375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.729716e+06' disk.nvme1c1n1.out‚1710.70361328125 disk.dm-1.in‚0! disk.dm-1.out‚930.92041015625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚3164.57373046875$ memory_percent‚95.38767337799072# proc.memory.rssMB‚ 2918.06640625 disk.md0.in‚0( proc.memory.availableMB‚ 755.68359375* proc.memory.percent‚17.810463905334473$ disk./.usageGB‚1170.035057067871 disk.nvme0n1.in‚0% disk.nvme0n1.out‚3164.57373046875 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1710.70361328125 disk.nvme0c0n1.in‚0 cpu‚25.865685396064492 network.recv‚273535) disk./.usagePercent‚76.21136994417007 disk.nvme1n1.in‚0 disk.md0.out‚ 19.85546875ºÁtʞׂ(:“ óìôÒ÷΄¿ gpu.0.gpu‚92.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚116.833' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.94433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô¬ô0Ø+ _wandbruntime‚847  _runtime‚847BVâ[ÅÙ¿ ' lr-AdamW/pg1‚0.00015970856411783474 ' lr-AdamW/pg2‚0.00015970856411783474 trainer/global_step‚1249 # _timestamp‚1.78449369077342e+09 _runtime‚ 847.376279475 _step‚99Bµh^ÉÚà # _timestamp‚1.78449369077342e+09 _runtime‚ 847.376279475 _step‚99 ' lr-AdamW/pg1‚0.00015970856411783474 ' lr-AdamW/pg2‚0.00015970856411783474 trainer/global_step‚1249c['O0Û+ _wandbruntime‚847  _runtime‚847žf÷Å$ÜÁO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97976412245504 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97976517944832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1061731517952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1027871212842 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6646062 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.f„ÆFH£reed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚164 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9905621 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12576095 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73689245223936 trainer/global_step‚1249 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9482919 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1027957999268 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905623 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12565952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1030786939904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2836853 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6646066 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22481718 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72627513705984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73689454097408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 _runtime‚ 847.376342555 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1030698279936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 _step‚100 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1030786939904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96945713965568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72627607815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2836689 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9905623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208873472 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96829849682330 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9482751 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95801995508848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12576095 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0»RMË'ÝÅO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96945731004928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9482751 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95801995508848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12576095 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97976412245504 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97976517944832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1061731517952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1027871212842 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6646062 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚164 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9905621 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12576095 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73689245223936 trainer/global_step‚1249 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9482919 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1027957999268 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905623 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12565952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1030786939904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2836853 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6646066 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22481718 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72627513705984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73689454097408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 _runtime‚ 847.376342555 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1030698279936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 _step‚100 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1030786939904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96945713965568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72627607815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2836689 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9905623 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 208873472 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96829849682330 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98052347627520 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚ 847.795859265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6651118 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1028741493980 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73746192816640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95876268994880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98052453334016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6651122 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2839076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1062654042112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9490018 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2838900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12575772 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1031572662784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73746399585792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97020863631872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 $ _timestamp‚1.784493691193562e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12575772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98052453334016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12585929 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12585929 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96905027528220 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97020863631872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9913250 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73746192816640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95876268994880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98052453334016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6651122 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2839076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1062654042112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9490018 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2838900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12575772 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1031572662784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73746399585792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97020863631872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 $ _timestamp‚1.784493691193562e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12575772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98052453334016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12585929 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12585929 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96905027528220 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97020863631872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9913250 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98052347627520 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚ 847.795859265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6651118 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048e ‘0á+ _wandbruntime‚847  _runtime‚847Óÿcâý .loss/FBt1/detection_loss‚3.085988392922904 _step‚102 0loss/FRt2/assignment_loss‚3.3156146429860978 1loss/SRqqt1/detection_loss‚0.8127954371498224 &loss/total_loss‚17.489520036060398 _runtime‚ 847.796721436 .loss/FBt2/detection_loss‚2.263224563021173 epoch‚3 % _timestamp‚1.7844936911950247e+09 0loss/FRt1/assignment_loss‚1.7792001810802776 2loss/SRqqt1/assignment_loss‚0.9216140385592144 0loss/FBt1/assignment_loss‚0.1223364886733197 trainer/global_step‚1249 2loss/SRqqt2/assignment_loss‚1.2601965673174893 0loss/FBt2/assignment_loss‚0.1613501472798999 /loss/FRt1/detection_loss‚1.3770949074525423 /loss/FRt2/detection_loss‚0.5796685984411751 1loss/SRqqt2/detection_loss‚1.8104360711764864ƒ/Gàã .loss/FBt2/detection_loss‚2.263224563021173 epoch‚3 % _timestamp‚1.7844936911950247e+09 0loss/FRt1/assignment_loss‚1.7792001810802776 2loss/SRqqt1/assignment_loss‚0.9216140385592144 0loss/FBt1/assignment_loss‚0.1223364886733197 trainer/global_step‚1249 2loss/SRqqt2/assignment_loss‚1.2601965673174893 0loss/FBt2/assignment_loss‚0.1613501472798999 /loss/FRt1/detection_loss‚1.3770949074525423 /loss/FRt2/detection_loss‚0.5796685984411751 1loss/SRqqt2/detection_loss‚1.8104360711764864 .loss/FBt1/detection_loss‚3.085988392922904 _step‚102 0loss/FRt2/assignment_loss‚3.3156146429860978 1loss/SRqqt1/detection_loss‚0.8127954371498224 &loss/total_loss‚17.489520036060398 _runtime‚ 847.796721436fu[‹hAä‚(:¶ ‚íôÒ¥š™¿* proc.memory.percent‚17.810559272766113 disk.md0.in‚0 disk.md0.out‚ 20.3828125$ memory_percent‚95.38943767547607# proc.memory.rssMB‚ 2918.08203125 network.recv‚276783' disk.nvme0c0n1.out‚5083.85595703125' disk.nvme1c1n1.out‚3624.87353515625 disk.nvme1n1.in‚0 disk.rbd0.in‚0 cpu‚25.862844685160958 proc.cpu.threads‚17% disk./.usageGB‚1175.5665588378906 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚3624.87353515625 disk.dm-1.in‚0! disk.dm-1.out‚931.08642578125 disk.rbd0.out‚0( proc.memory.availableMB‚ 755.39453125 network.sent‚ 1.745523e+06) disk./.usagePercent‚76.57166968493017 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚5083.85595703125 disk.dm-0.in‚0" disk.dm-0.out‚7733.52978515625ø •¢žå‚(:“ ‚íôÒÅ÷‹¿ gpu.0.gpu‚76.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.252' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.41733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á1t;æ‚(:° ‘íôÒÇჿ% disk./.usageGB‚1174.0260963439941 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚6410.7822265625 disk.nvme1c1n1.in‚0* proc.memory.percent‚17.810559272766113 cpu‚25.858960158816124% disk.nvme1n1.out‚4039.97021484375 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚4039.97021484375( proc.memory.availableMB‚ 755.64453125) disk./.usagePercent‚76.47133016407713 disk.nvme1n1.in‚0 disk.dm-0.out‚ 9474.765625! disk.dm-1.out‚931.35205078125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.794971e+06 network.recv‚280962 disk.md0.in‚0 disk.md0.out‚ 20.62109375 disk.dm-0.in‚0& disk.nvme0c0n1.out‚6410.7822265625$ memory_percent‚95.38791179656982# proc.memory.rssMB‚ 2918.08203125™/¬.žç‚(:“ ‘íôÒʉ¿ gpu.0.gpu‚90.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.943' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.98100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÙƒ`0è+ _wandbruntime‚879  _runtime‚879ø4îÎÈé ' lr-AdamW/pg2‚0.00015961878882926798 trainer/global_step‚1299 % _timestamp‚1.7844937224577882e+09 _runtime‚ 879.059614044 _step‚103 ' lr-AdamW/pg1‚0.00015961878882926798‘B…^ÌêÆ trainer/global_step‚1299 % _timestamp‚1.7844937224577882e+09 _runtime‚ 879.059614044 _step‚103 ' lr-AdamW/pg1‚0.00015961878882926798 ' lr-AdamW/pg2‚0.00015961878882926798güÄÛ0ë+ _wandbruntime‚879  _runtime‚8793ODÕ'ìÏO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1070048220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1102500414464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2946205 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1067109743650 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100703197280768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213067776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1067196530076 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75434400105984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101773351200256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13058804 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101773245500928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _runtime‚ 879.060434336 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23345875 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100703214320128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1102381456384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13068947 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101773351200256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76536781562368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99516504739088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10287073 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118958080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9845067 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 % _timestamp‚1.7844937224586804e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1070136880128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100583718308524 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6898862 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100703197280768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10287071 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9845255 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100583614482738 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13068947 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101773245500928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2946205 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1067109743650 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100703197280768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213067776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1067196530076 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75434400105984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101773351200256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13058804 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101773245500928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _runtime‚ 879.060434336 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23345875 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100703214320128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1102381456384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13068947 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101773351200256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76536781562368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99516504739088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10287073 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118958080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9845067 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 % _timestamp‚1.7844937224586804e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1070136880128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100583718308524 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6898862 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100703197280768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10287071 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9845255 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100583614482738 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13068947 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101773245500928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100778363986432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2948364 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99590795264480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101849286589440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100778346947072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9852282 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23363324 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13068624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1070833935872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10294700 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1103194466816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_tst¼ñxrain_batch_end/requested_bytes.all.freed‚100658688502730 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76593732300800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6903918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99590812303840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75490631943680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13068624 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6903922 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100778346947072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚105 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1070922603008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23373483 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76593953750016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10294700 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101849180882944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10294702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100658792328628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1067980024788 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 879.463670757 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1070833935872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6903922 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100778346947072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚105 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1070922603008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23373483 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76593953750016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10294700 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101849180882944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10294702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100658792328628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1067980024788 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 879.463670757 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1070833935872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100778363986432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2948364 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99590795264480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101849286589440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100778346947072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9852282 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23363324 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13068624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1070833935872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10294700 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1103194466816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100658688502730 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76593732300800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6903918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99590812303840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75490631943680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13068624 BAõ‚(:’  íôÒÒí¿ gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚219.749' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.24966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DP((ö‚(: ¯íôÒï²é¾ disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚6429.861328125* proc.memory.percent‚17.810630798339844 cpu‚25.849377432666188# disk.nvme1c1n1.out‚ 4043.2734375 disk.dm-1.in‚0 disk.dm-1.out‚ 931.65625 disk.nvme0c0n1.in‚0# memory_percent‚95.3876256942749 proc.cpu.threads‚17 disk.dm-0.out‚9495.619140625 network.recv‚288363 disk.nvme0n1.in‚0# disk.nvme0n1.out‚6429.861328125! disk.nvme1n1.out‚ 4043.2734375 disk.md0.in‚0 disk.md0.out‚ 21.17578125( proc.memory.availableMB‚ 755.69140625' disk./.usagePercent‚76.471327430863% disk./.usageGB‚1174.0260543823242 disk.nvme1n1.in‚0 proc.memory.rssMB‚ 2918.09375 disk.dm-0.in‚0 network.sent‚ 1.863864e+06 disk.nvme1c1n1.in‚0=³€½ž÷‚(:“ ¯íôÒ¤©˜¿ gpu.0.gpu‚91.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚224.426' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.80866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»©»q0ø+ _wandbruntime‚909  _runtime‚909³£ÇùÁ _runtime‚ 909.996478777 _step‚107 ' lr-AdamW/pg1‚0.00015952391423365087 ' lr-AdamW/pg2‚0.00015952391423365087 trainer/global_step‚1349 $ _timestamp‚1.784493753394646e+09ǾËúÅ ' lr-AdamW/pg1‚0.00015952391423365087 ' lr-AdamW/pg2‚0.00015952391423365087 trainer/global_step‚1349 $ _timestamp‚1.784493753394646e+09 _runtime‚ 909.996478777 _step‚107kj›Ó0û+ _wandbruntime‚910  _runtime‚910·š2²Ù'üÓO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7151662 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3054103 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104460697635328 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1109398160384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104460680595968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78241286505984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1106348274458 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105570184455680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79384225653248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1143066493952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1109486820352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13561799 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10205769 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104337379283146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7151666 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105570078756352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105570078756352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 %œyŸý×O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79384447109632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104337483108932 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103231031008688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10205582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105570078756352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7151662 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3054103 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 `LDeviceStatsMonitor.on_train_§ü¨>!batch_start/allocated_bytes.large_pool.allocated‚104460697635328 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1109398160384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104460680595968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78241286505984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1106348274458 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105570184455680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79384225653248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1143066493952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1109486820352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13561799 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10205769 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104337379283146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7151666 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105570078756352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79441173090304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13571633 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7156718 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 % _timestamp‚1.7844937537945075e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3056298 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676150 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104412453303138 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1107218555596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79441390345216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1143872001536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103305338573440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1110272543232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78297518343680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 910.396460467 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104535830262272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10212842 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24237626 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104412557129036 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _step‚109 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105646014138368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104535830262272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10213020 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚1349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3056124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0à "ç&€áM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 910.396460467 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104535830262272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10212842 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24237626 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104412557129036 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _step‚109 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105646014138368 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104535830262272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10213020 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚1349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3056124 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24247785 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10676150 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105646119844864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104535847301632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1110183876096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78297424233984 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79441173090304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13571633 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7156718 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 % _timestamp‚1.7844937537945075e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3056298 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676150 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104412453303138 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1107218555596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79441390345216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1143872001536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103305338573440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1110272543232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78297518343680 F†‚(:³ ÍíôÒÕ­Þ¾ disk.dm-1.in‚0 disk.dm-1.out‚ 931.869140625( proc.memory.availableMB‚ 755.89453125 cpu‚25.840926696736084 network.recv‚295070 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2918.09765625 proc.cpu.threads‚17) disk./.usagePercent‚76.45584973619613% disk./.usageGB‚1173.7884330749512 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 21.80859375* proc.memory.percent‚17.810654640197754 network.sent‚ 1.930121e+06 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚6443.27099609375& disk.nvme1c1n1.out‚4045.3837890625" disk.dm-0.out‚9509.53759765625$ memory_percent‚95.38638591766357% disk.nvme0n1.out‚6443.27099609375$ disk.nvme1n1.out‚4045.3837890625gž‡‚(:“ ÍíôÒÕœ’¿ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚122.756' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.91866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜YW0ˆ+ _wandbruntime‚941  _runtime‚941´$ŽS‰À trainer/global_step‚1399 % _timestamp‚1.7844937846119006e+09 _runšÔocstime‚ 941.213732415 _step‚111 & lr-AdamW/pg1‚0.0001594239464246014 & lr-AdamW/pg2‚0.0001594239464246014¤õ ´ÊŠÄ % _timestamp‚1.7844937846119006e+09 _runtime‚ 941.213732415 _step‚111 & lr-AdamW/pg1‚0.0001594239464246014 & lr-AdamW/pg2‚0.0001594239464246014 trainer/global_step‚1399o7Dì0‹+ _wandbruntime‚941  _runtime‚941pßØ'ŒÒO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109366912011776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 _runtime‚ 941.214551006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1148836760576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109367017711104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11049973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106945557278288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14044508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7404462 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81048172905984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108091144083554 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14054651 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109366912011776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7404466 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14044508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1183635309568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11049973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14054651 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1399 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108218180950528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3161722 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108091247909340 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1145673591692 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1145586805266 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11049971 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚108218180950528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109367017711104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10566184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3161903 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1148748100608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81048267015680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1183503768576 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 $ _timestamp‚1.784493784612842e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108218163911168 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1148836760576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10566369 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25104624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82231676674560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130C—rÜ'ÖO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚108218163911168 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81048172905984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108091144083554 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14054651 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109366912011776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7404466 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14044508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1183635309568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11049973 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14054651 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚1399 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108218180950528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3161722 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108091247909340 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1145673591692 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1145586805266 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11049971 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚108218180950528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109367017711104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10566184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3161903 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1148748100608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81048267015680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1183503768576 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 $ _timestamp‚1.784493784612842e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108218163911168 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1148836760576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10566369 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25104624 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82231676674560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109366912011776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 _runtime‚ 941.214551006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1148836760576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109367017711104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11049973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106945557278288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14044508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7404462 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10573452 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚173 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82288846363648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11057602 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107019847803680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 % _timestamp‚1.7844937850126097e+09 _step‚113 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10573629 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11057602 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1146457086404 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25122087 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108293313577472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109442847393792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108166321929444 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1146370299866 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108293330616832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108293330616832 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82288629108736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1184441619968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14064485 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11057600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1149533816320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1149622483456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25111928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11057600 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14054328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7409522 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1184318474752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25111928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1149622483456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14064485 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107019864843040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1149533816320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108293313577472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81104310633984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10573452 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚173 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82288846363648 REDeviceStatsMonitor.on_train_batch_end/allocation.laBZ¢ðSrge_pool.allocated‚11057602 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107019847803680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 % _timestamp‚1.7844937850126097e+09 _step‚113 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10573629 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11057602 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1146457086404 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25122087 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108293313577472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109442847393792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108166321929444 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1146370299866 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108293330616832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108293330616832 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82288629108736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1184441619968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14064485 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11057600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1149533816320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1149622483456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25111928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11057600 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14054328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7409522 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1184318474752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25111928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1149622483456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14064485 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107019864843040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1149533816320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108293313577472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81104310633984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113163745267200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14537360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1188186700800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85079333179392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110660100587248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1188098040832 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10926411 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184912122500 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11431423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1188098040832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7657262 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83855059305984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113163850966528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10926588 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111845012709748 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25968781 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111975664265728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184825336074 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14547503 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 % _timestamp‚1.7844938161986003e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111975647226368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3269322 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1224179763712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111975647226368 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85079113820160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚116 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25978926 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111844908883962 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110660083547888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111975664265728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11431423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1224054514176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14547503 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1188186700800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 _runtime‚ 972.800335039 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7657266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111844908883962 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110660083547888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111975664265728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11431423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1224054514176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14547503 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1188186700800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 _runtime‚ 972.800335039 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7657266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113163745267200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14537360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1188186700800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85079333179392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110660100587248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1188098040832 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10926411 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184912122500 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11431423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1188098040832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7657262 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83855059305984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113163850966528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10926588 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111845012709748 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25968781 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111975664265728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184825336074 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14547503 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 % _timestamp‚1.7844938161986003e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111975647226368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3269322 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1224179763712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_t¸ñŽArain_batch_start/allocated_bytes.large_pool.freed‚111975647226368 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85079113820160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚116tF´gÄ0ž+ _wandbruntime‚973  _runtime‚973•RyGã&ŸÝM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1188883756544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113239786355712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3271474 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11439050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188972423680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14547180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83911291143680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1224865427456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3271304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚1449 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7662318 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83911197033984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10933622 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25996389 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188972423680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113239680649216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25986230 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _step‚117 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112050796892672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188883756544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14557337 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1224994864128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110734391112640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112050813932032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110734374073280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1185608830674 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10933796 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14557337 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11439052 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112050813932032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112050796892672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1185695617212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10933622 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25996389 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188972423680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113239680649216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25986230 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _step‚117 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112050796892672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188883756544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14557337 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1224994864128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110734391112640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112050813932032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110734374073280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1185608830674 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10933796 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14557337 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11439052 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112050813932032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112050796892672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1185695617212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11439050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188972423680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14547180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83911291143680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1224865427456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3271304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚1449u©.=Ÿ0¡+ _wandbruntime‚973  _runtime‚973´¾Ë©¢þ epoch‚3 _runtime‚ 973.202419725 0loss/FRt2/assignment_loss‚3.1443860482969446 2loss/SRqqt2/assignment_loss‚1.1874531997283073 .loss/FRt2/detection_loss‚0.563333649691211 /loss/FBt1/detection_loss‚1.7539247742278894 /loss/FRt1/detection_loss‚1.3946068824962017 &loss/total_loss‚15.533211818086627 % _timestamp‚1.7844938166008742e+09 1loss/FBt1/assignment_loss‚0.04406777720543023 0loss/FBt2/assignment_loss‚0.1435405032795908 1loss/SRqqt2/detection_loss‚1.6703220023817946 .loss/FBt2/detection_loss‚2.495389836668009 trainer/global_step‚1449 _step‚118 0loss/FRt1/assignment_loss‚1.4858748580043148 2loss/SRqqt1/assignment_loss‚0.9274749272365652 1loss/SRqqt1/detection_loss‚0.7228373588703693®¹]•£‚ &loss/total_loss‚15.533211818086627 % _timestamp‚1.7844938166008742e+09 1loss/FBt1/assignment_loss‚0.04406777720543023 0loss/FBt2/assignment_loss‚0.1435405032795908 1loss/SRqqt2/detection_loss‚1.6703220023817946 .loss/FBt2/detection_loss‚2.495389836668009 trainer/global_step‚1449 _step‚118 0loss/FRt1/assignment_loss‚1.4858748580043148 2loss/SRqqt1/assignment_loss‚0.9274749272365652 1loss/SRqqt1/detection_loss‚0.7228373588703693 epoch‚3 _runtime‚ 973.202419725 0loss/FRt2/assignment_loss‚3.1443860482969446 2loss/SRqqt2/assignment_loss‚1.1874531997283073 .loss/FRt2/detection_loss‚0.563333649691211 /loss/FBt1/detection_loss‚1.7539247742278894 /loss/FRt1/detection_loss‚1.3946068824962017vÇ©`Ê:¤‚(:¯ úíôÒö踿 network.recv‚306396) disk./.usagePercent‚76.45588526797982 disk.dm-0.in‚0 disk.dm-1.in‚0& disk.nvme1c1n1.out‚4077.3056640625 disk.nvme1n1.in‚0 disk.md0.in‚0( proc.memory.availableMB‚ 690.34765625! disk.dm-1.out‚932.30810546875 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚4077.3056640625 disk.md0.out‚ 22.8984375# proc.memory.rssMB‚ 2918.85546875* proc.memory.percent‚17.815279960632324 cpu‚25.85646920469525% disk./.usageGB‚1173.7889785766602 disk.dm-0.out‚9780.197265625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6684.79638671875 network.sent‚ 2.014149e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚6684.79638671875" memory_percent‚95.786452293396 proc.cpu.threads‚17ÓfÍWœ¥‚(:‘ úíôÒ¹«å¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.583' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.19433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mðxž¦‚(:“ ‰îôÒ”ç¾ gpu.0.gpu‚91.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚124.759' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.58633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•tÑ8§‚(:­ ‰îôÒس̿) disk./.usagePercent‚76.47585711209747 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 network.sent‚ 2.064631e+06" disk.nvme1n1.out‚ 4097.90234375 disk.dm-0.out‚ 9804.5703125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚6689.29736328125* proc.memory.percent‚17.810654640197754 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 23.19140625 disk.dm-1.in‚0! disk.dm-1.out‚932.40185546875 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 691.09765625 cpu‚25.858605890145476 network.recv‚310918% disk./.usageGB‚1174.0955963134766% disk.nvme0n1.out‚6689.29736328125$ disk.nvme1c1n1.out‚ 4097.90234375$ memory_percent‚95.78187465667725# proc.memory.rssMB‚ 2918.09765625 proc.cpu.threads‚17ar¹U2¨- _wandbruntime‚1004  _runtime‚1004RÆ©À $ _timestamp‚1.784493847785121e+09 _runtime‚1004.386950785 _step‚119 & lr-AdamW/pg1‚0.0001592087571759029 & lr-AdamW/pg2‚0.0001592087571759029 trainer/global_step‚1499jjOJ滀 trainer/global_step‚1499 $ _timestamp‚1.784493847785121e+09 _runtime‚1004.386950785 _step‚119 & lr-AdamW/pg1‚0.0001592087571759029 & lr-AdamW/pg2‚0.0001592087571759029w 2«- _wandbruntime‚1004  _runtime‚1004Ì ×fy¬ÔO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114374626856848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115733130541568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115733130541568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86662039815680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116960578522624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116960578522624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1264694892544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1224063866882 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11812871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116960684221952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115733147580928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3376783 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86661945705984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115598777510156 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115733147580928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _step‚120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11286849 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87926502766080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11812873 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844938477859952e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚1004.387680905 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15030212 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1227447981056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137832448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1227447981056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26853228 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15040355 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115598673684370 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7910062 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26843083 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1227536641024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/inactive_splitÒçha .all.freed‚11286660 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1227536641024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87926734708224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264557060096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0$ÿÞ'­ØO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11812873 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11812871 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1224150653308 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114374609817488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264557060096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114374626856848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115733130541568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115733130541568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86662039815680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116960578522624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116960578522624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1264694892544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1224063866882 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11812871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116960684221952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115733147580928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3376783 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86661945705984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115598777510156 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115733147580928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _step‚120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11286849 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87926502766080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11812873 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844938477859952e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚1004.387680905 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15030212 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1227447981056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137832448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1227447981056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26853228 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15040355 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115598673684370 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7910062 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26843083 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1227536641024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11286660 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1227536641024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87926734708224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15050189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1228233696768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11820500 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11820500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11293834 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117036619611136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114448917382240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1228322363904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3378903 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115673747704362 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1265496033792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26860532 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11820502 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15040032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117036513904640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115808297247232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117036619611136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7915118 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115808297247232 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87983450031104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 _step‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87983673577472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114448900342880 % _timestamp‚1.7844938481864498e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11820502 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11294025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15040032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1228322363904éæ”Ôè&°âM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115673851530260 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117036619611136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7915118 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115808297247232 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87983450031104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 _step‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87983673577472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114448900342880 % _timestamp‚1.7844938481864498e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11820502 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11294025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15040032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1228322363904 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26860532 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15050189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1228233696768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11820500 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11820500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11293834 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117036619611136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114448917382240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1228322363904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3378903 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115673747704362 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1265496033792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26860532 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11820502 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15040032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117036513904640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115808297247232 Fó°Ì¼Æ _step‚123 ' lr-AdamW/pg1‚0.00015909354955745717 ' lr-AdamW/pg2‚0.00015909354955745717 trainer/global_step‚1549 $ _timestamp‚1.784493879254748e+09 _runtime‚1035.856643977{píþP2½- _wandbruntime‚1035  _runtime‚1035OWæßÚ'¾ÔO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120757411778048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1305104634880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8162866 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118089153126448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3484372 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8162862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119352438484778 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119490613856768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1266797921280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12194321 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119490630896128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1266886581248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1305227787264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118089136087088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120757411778048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12194323 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1263389184116 _step‚124 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚167 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89468832105984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1266797921280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11647238 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120757517477376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11647067 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119490630896128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90773936740864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1035.859152352‹ðÛÞ'¿ØO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12194323 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1263389184116 _step‚124 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚167 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89468832105984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1266797921280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11647238 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120757517477376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11647067 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119490630896128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90773936740864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1035.859152352 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120757411778048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1305104634880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8162866 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118089153126448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3484372 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8162862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119352438484778 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119490613856768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1266797921280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12194321 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119490630896128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1266886581248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1305227787264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118089136087088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120757411778048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786|sÏe2À- _wandbruntime‚1036  _runtime‚1036öd U‘ÁÞM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1267672304128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1305912115712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1264172678828 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119565763523072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89525063943680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚164 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120833452866560 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12201950 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3486470 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12201952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11654228 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89524969833984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120833452866560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119565763523072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119427616330668 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1267583636992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚°(”½S 160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1267583636992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90830881949696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119427512504770 trainer/global_step‚1549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11654392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15543041 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _runtime‚1036.263716924 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15543041 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15532884 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1264085892290 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760±Êêè&ÂâM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1267672304128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119565780562432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12201952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11654228 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89524969833984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120833452866560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119565763523072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119427616330668 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1267583636992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1267583636992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90830881949696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119427512504770 trainer/global_step‚1549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11654392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15543041 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _runtime‚1036.263716924 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15543041 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15532884 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1264085892290 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1267672304128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1305912115712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1264172678828 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119565763523072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89525063943680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚164 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120833452866560 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12201950 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3486470 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BÎòÚ'ÎÔO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1306147861504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124554245033472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚1599 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16015916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12006392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124554245033472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12575771 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28601832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1306236521472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12575771 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93621231692288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8415666 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123106307110972 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16026059 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123248097171968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123248114211328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93621453148672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124554350732800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92275812615680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121803679396048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123248114211328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8415662 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3590910 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3590730 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1345640532992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1306147861504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16026059 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121803662356688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123106203285186 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12575773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12575773 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1302540928498 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124554350732800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1306236521472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12006576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345513186304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _step‚128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 _runtime‚1067.172067958éx ‡’ÏÙO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8415666 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123106307110972 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16026059 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123248097171968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123248114211328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93621453148672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124554350732800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92275812615680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121803679396048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123248114211328 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8415662 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3590910 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3590730 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1345640532992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1306147861504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16026059 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121803662356688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123106203285186 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12575773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12575773 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1302540928498 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VBDeviceStatsMonitorÊ¢C’M.on_train_batch_start/active_bytes.all.allocated‚124554350732800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1306236521472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12006576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345513186304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _step‚128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 _runtime‚1067.172067958 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1306147861504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124554245033472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚1599 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16015916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12006392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124554245033472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12575771 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28601832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1306236521472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12575771 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93621231692288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552€¾¸D2Ð- _wandbruntime‚1067  _runtime‚1067¡î÷ïâ&ÑÜM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1306933577216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121877969921440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123181381131076 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12583400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844939109707527e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12583402 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3592825 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93678402831360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _step‚129 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1303411209636 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123323246838272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28619295 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3592996 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12583402 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16035893 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92331856233984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124630286121984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8420718 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12013543 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93678175090688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12013718 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121877952882080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16025736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12583400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844939109707527e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12583402 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3592825 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93678402831360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _step‚129 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1303411209636 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123323246838272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚171 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28619295 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3592996 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12583402 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16035893 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92331856233984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124630286121984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8420718 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12013543 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93678175090688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12013718 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121877952882080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16025736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1830623298883438 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_2_of_3‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19816072285175323 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8643949627876282 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1830623298883438 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5467851758003235 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02899041399359703 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28455811738967896 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9889604449272156 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23906859755516052 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5059269070625305 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/accuracy‚0.828252911567688 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02627013996243477 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5463624596595764 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.07585455477237701 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9931479096412659 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.8948591947555542 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9782994389533997 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2870370149612427 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5288649797439575 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5963954925537109 ) Purity/0FRt1SRqqt2FBt/FBt_purity‚0.75 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 jet/accuracy_4_of_5‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018036523833870888 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *jet/accuracy_1_of_2‚0.7355175614356995 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.07735518366098404 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.506633460521698 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6284053921699524 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4191667139530182 /particle/accuracy_4_of_4‚0.9247377514839172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4287084937095642 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.17787285149097443 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Pu’Ä­-rity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5074057579040527 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 _runtime‚1080.951551711 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2870370149612427 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6880196332931519 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23670898377895355 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20610098540782928 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5765226483345032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23303039371967316 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19286540150642395 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7052721381187439 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2154899537563324 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.21875 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23300115764141083 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4765625298023224 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3046875 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.672846257686615 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.991786777973175 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2237934172153473 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17013049125671387 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9925667643547058 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.29522839188575745 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.48191696405410767 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.36609703302383423 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996975839138031 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.22843529284000397 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4927155077457428 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013368410989642143 *jet/accuracy_1_of_4‚0.9470576047897339 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8148148059844971 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8134313225746155 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.96778404712677 trainer/global_step‚1599 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,jet/accuracy_3_of_4‚0.048541851341724396 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5321246385574341 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6010758280754089 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9706915616989136 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6285160779953003 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5575774312019348 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4336715340614319 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6334323287010193 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13916152715682983 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4380134046077728 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_3_of_3‚0.052411999553442 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4852323532104492 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5770614147186279 +jet/accuracy_1_of_1‚0.27022939920425415 /particle/accuracy_3_of_4‚0.9970238208770752 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.36942043900489807 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9913819432258606 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9951900243759155 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.47809088230133057 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6017621755599976 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9806329011917114 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9741911888122559 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.30027374625205994 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5985033512115479 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.991786777973175 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19151601195335388 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_3‚0.9990828633308411 particle/accuracy_2_of_4‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3009259104728699 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5869171023368835 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7018707394599915 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4327673316001892 +jet/accuracy_2_of_2‚0.09902256727218628 particle/accuracy_2_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9197685718536377 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6122313737869263 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5117157101631165 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19366233050823212 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20812147855758667 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7316845059394836 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 5validation_average_jet_accuracy‚0.313418984413147 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _step‚131 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2889455258846283 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.688359797000885 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22939345240592957 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9774295091629028 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17590241134166718 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23447711765766144 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5000515580177307 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.600645899772644 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9321974515914917 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5898127555847168 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4642857015132904 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938583374023438 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1836266815662384 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5349839925765991 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.1725662499666214 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.006769374944269657 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4957248568534851 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.600645899772644 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21970608830451965 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7833216786384583 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24599763751029968 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19606682658195496 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23526602983474731 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46875 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18203851580619812 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14687423408031464 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22455613315105438 particle/accuracy_5_of_5‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_3‚1 *particle/sensitivity‚0.584701657295227 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2436748743057251 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5979081392288208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2955615222454071 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48787885904312134 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7844939243491068e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037356819957494736 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9877369999885559 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9636048674583435 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22843529284000397 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9473453760147095 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9750000238418579 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666728258133 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8075358867645264 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_2_of_4‚0.6707976460456848 jet/accuracy_3_of_5‚1 *loss/val_total_loss‚14.165369669954172 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8583294153213501 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9732218384742737mÆÙ=ÚÓ{ ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018036523833870888 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *jet/accuracy_1_of_2‚0.7355175614356995 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.07735518366098404 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.506633460521698 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6284053921699524 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4191667139530182 /particle/accuracy_4_of_4‚0.9247377514839172 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4287084937095642 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.17787285149097443 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5074057579040527 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 _runtime‚1080.951551711 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2870370149612427 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6880196332931519 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23670898377895355 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20610098540782928 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5765226483345032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23303039371967316 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19286540150642395 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7052721381187439 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2154899537563324 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.21875 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.23300115764141083 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4765625298023224 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3046875 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.672846257686615 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.991786777973175 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2237934172153473 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17013049125671387 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9925667643547058 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.29522839188575745 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.48191696405410767 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.36609703302383423 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996975839138031 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.22843529284000397 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4927155077457428 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013368410989642143 *jet/accuracy_1_of_4‚0.9470576047897339 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8148148059844971 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8134313225746155 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.96778404712677 trainer/global_step‚1599 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,jet/accuracy_3_of_4‚0.048541851341724396 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5321246385574341 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6010758280754089 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9706915616989136 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6285160779953003 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5575774312019348 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4336715340614319 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6334323287010193 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13916152715682983 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4380134046077728 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_3_of_3‚0.052411999553442 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4852323532104492 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5770614147186279 +jet/accuracy_1_of_1‚0.27022939920425415 /particle/accuracy_3_of_4‚0.9970238208770752 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.36942043900489807 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9913819432258606 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9951900243759155 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.47809088230133057 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6017621755599976 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9806329011917114 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9741911888122559 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.30027374625205994 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5985033512115479 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.991786777973175 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19151601195335388 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_3‚0.9990828633308411 particle/accuracy_2_of_4‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3009259104728699 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5869171023368835 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7018707394599915 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4327673316001892 +jet/accuracy_2_of_2‚0.09902256727218628 particle/accuracy_2_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9197685718536377 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6122313737869263 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5117157101631165 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19366233050823212 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20812147855758667 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7316845059394836 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 5validation_average_jet_accuracy‚0.313418984413147 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _step‚131 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2889455258846283 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.688359797000885 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22939345240592957 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9774295091629028 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17590241134166718 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.23447711765766144 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5000515580177307 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.600645899772644 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9321974515914917 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5898127555847168 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4642857015132904 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938583374023438 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1836266815662384 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5349839925765991 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.1725662499666214 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.006769374944269657 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4957248568534851 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.600645899772644 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21970608830451965 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7833216786384583 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24599763751029968 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19606682658195496 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23526602983474731 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46875 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18203851580619812 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14687423408031464 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22455613315105438 particle/accuracy_5_of_5‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_3‚1 *particle/sensitivity‚0.584701657295227 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2436748743057251 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5979081392288208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2955615222454071 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48787885904312134 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7844939243491068e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.037356819957494736 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9877369999885559 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9636048674583435 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22843529284000397 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9473453760147095 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9750000238418579 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9851190447807312 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666728258133 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8075358867645264 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_2_of_4‚0.6707976460456848 jet/accuracy_3_of_5‚1 *loss/val_total_loss‚14.165369669954172 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8583294153213501 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9732218384742737 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009223628789186478 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9364289045333862 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5000515580177307 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.3931651711463928 jet/accuracy_2_of_5‚1 epoch‚3 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4917748272418976 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5292701721191406 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_3‚0.8775017857551575 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3935185372829437 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5421849489212036 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.46627381443977356 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5883896946907043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_2‚1 'particle/f_score‚0.5938690304756165 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2549874484539032 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.48368924856185913 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1830623298883438 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_2_of_3‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19816072285175323 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8643949627876282 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1830623298883438 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5467851758003235 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02899041399359703 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28455811738967896 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9889604449272156 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23906859755516052 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5059269070625305 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/accuracy‚0.828252911567688 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02627013996243477 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5463624596595764 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.07585455477237701 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9931479096412659 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.8948591947555542 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9782994389533997 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2870370149612427 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5288649797439575 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5963954925537109 ) Purity/0FRt1SRqqt2FBt/FBt_purity‚0.75 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 jet/accuracy_4_of_5‚0ƒÄÜF„>Û‚(:³ òîôÒ°ŠÐ¾ disk.nvme0n1.in‚0$ memory_percent‚95.89700698852539) proc.memory.percent‚17.92922019958496 proc.cpu.threads‚17 network.recv‚337753) disk./.usagePercent‚76.55791963006712% disk./.usageGB‚1175.3554611206055' disk.nvme0c0n1.out‚7424.35595703125& proc.memory.availableMB‚ 672.234375 cpu‚25.876325040915003 network.sent‚ 2.27007e+06 disk.md0.in‚0 disk.dm-0.in‚0# disk.dm-0.out‚11603.20849609375 disk.rbd0.out‚0% disk.nvme0n1.out‚7424.35595703125 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2937.5234375% disk.nvme1n1.out‚5233.39306640625 disk.md0.out‚ 26.19921875 disk.dm-1.in‚0' disk.nvme1c1n1.out‚5233.39306640625 disk.nvme1n1.in‚0! disk.dm-1.out‚997.88623046875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 ›³ Ü‚(:’ òîôÒ×¼ƒÀ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.525' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.175000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•%‘¯9Ý‚(:® ïôÒò¬™¿ disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.dm-0.out‚ 11612.0078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚7431.8349609375$ memory_percent‚95.89121341705322" proc.memory.rssMB‚ 2936.5703125 cpu‚25.873721632176142% disk./.usageGB‚1175.2985649108887% disk.nvme1n1.out‚5299.16650390625 disk.dm-0.in‚0" disk.dm-1.out‚1061.81591796875& disk.nvme0c0n1.out‚7431.8349609375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5299.16650390625* proc.memory.percent‚17.923402786254883 network.sent‚ 2.327825e+06) disk./.usagePercent‚76.55421364018164 disk.md0.out‚26.4375( proc.memory.availableMB‚ 673.18359375 proc.cpu.threads‚17 network.recv‚341814 disk.nvme1n1.in‚0 disk.md0.in‚0GÅꛓÞ‚(:ˆ ïôÒ»íÊ¿ gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚219.606' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£.eÏ2ß- _wandbruntime‚1112  _runtime‚1112Áôo#Éàà ' lr-AdamW/pg1‚0.00015884794532974782 ' lr-AdamW/pg2‚0.00015884794532974782 trainer/global_step‚1649 % _timestamp‚1.7844939558161771e+09 _runtime‚1112.418781716 _step‚132Ä—£çÎáÈ % _timestamp‚1.7844939558161771e+09 _runtime‚1112.418781716 _step‚132 ' lr-AdamW/pg1‚0.00015884794532974782 ' lr-AdamW/pg2‚0.00015884794532974782 trainer/global_step‚1649„縘Ã2â- _wandbruntime‚1112  _runtime‚1112õ.‹ãÔO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13079023 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16591275 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 114763776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128002084822528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844939558169525e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079021 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129363100661248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129362994961920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128002067783168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12521518 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 trainer/global_step‚1649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29670296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8776717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29680441 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079023 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3744801 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95885970110976 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126491655662480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97287653965824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1360927178752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8776721 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401683854848 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127848954206138 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13079021 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚128002084822528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1357281504298 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128002067783168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95886064220672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1401798618624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126491672701840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12521701 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16601418 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95886064220672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1401798618624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126491672701840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12521701 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16601418 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129362994961920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128002067783168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12521518 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 trainer/global_step‚1649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29670296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8776717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29680441 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079023 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3744801 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95885970110976 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126491655662480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97287653965824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1360927178752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8776721 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401683854848 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127848954206138 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13079021 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960…yòÙ2å- _wandbruntime‚1112  _runtime‚1112ú]Çã&æÝM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97344811186688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126565946187872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16601095 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3747156 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29687745 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126565963227232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1357978212472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128077234488832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1361801561600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12528748 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _runtime‚1112.821336107 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1361801561600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1361712894464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12528933 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129439036050432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚1649 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8781777 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127924028226242 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 $ _timestamp‚1.784493956219328e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13086652 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8781773 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97344600223232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128077234488832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16611252 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3746975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402609238016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402492384256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1358064999010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29687745 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16611252 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129439036050432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128077217449472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128077217449472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95942201948672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129438930343936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95942107838976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16601095 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1361712894464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488\‡ojšçâM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚1649 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8781777 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127924028226242 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 $ _timestamp‚1.784493956219328e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13086652 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8781773 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97344600223232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128077234488832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16611252 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3746975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402609238016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402492384256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1358064999010 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29687745 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16611252 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129439036050432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128077217449472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128077217449472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95942201948672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129438930343936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_spliæŽñàNt.small_pool.peak‚486 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95942107838976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16601095 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1361712894464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97344811186688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126565946187872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16601095 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3747156 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29687745 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126565963227232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1357978212472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128077234488832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1361801561600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12528748 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _runtime‚1112.821336107 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1361801561600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1361712894464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12528933 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129439036050432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736†ž¾§2è- _wandbruntime‚1112  _runtime‚1112Ô†ä-éý 2loss/SRqqt2/assignment_loss‚0.9379366931398628 0loss/FBt1/assignment_loss‚0.1078798173926932 /loss/FRt2/detection_loss‚0.6142768280103825 1loss/SRqqt1/detection_loss‚0.7802947986006624 1loss/SRqqt2/detection_loss‚1.3707475228804917 _runtime‚1112.821833738 0loss/FRt2/assignment_loss‚3.0365999032637316 /loss/FBt2/assignment_loss‚0.227607343507338 /loss/FBt1/detection_loss‚1.3233471568151238 /loss/FBt2/detection_loss‚1.6250147453529433 %loss/total_loss‚13.70284564126087 0loss/FRt1/assignment_loss‚1.5578424844575758 epoch‚4 trainer/global_step‚1649 _step‚135 1loss/SRqqt1/assignment_loss‚0.813379042007521 /loss/FRt1/detection_loss‚1.3079193058325431 % _timestamp‚1.7844939562202609e+09ŒP¤lê‚ /loss/FBt2/assignment_loss‚0.227607343507338 /loss/FBt1/detection_loss‚1.3233471568151238 /loss/FBt2/detection_loss‚1.6250147453529433 %loss/total_loss‚13.70284564126087 0loss/FRt1/assignment_loss‚1.5578424844575758 epoch‚4 trainer/global_step‚1649 _step‚135 1loss/SRqqt1/assignment_loss‚0.813379042007521 /loss/FRt1/detection_loss‚1.3079193058325431 % _timestamp‚1.7844939562202609e+09 2loss/SRqqt2/assignment_loss‚0.9379366931398628 0loss/FBt1/assignment_loss‚0.1078798173926932 /loss/FRt2/detection_loss‚0.6142768280103825 1loss/SRqqt1/detection_loss‚0.7802947986006624 1loss/SRqqt2/detection_loss‚1.3707475228804917 _runtime‚1112.821833738 0loss/FRt2/assignment_loss‚3.0365999032637316‡?(JÝ8ë‚(:­ ïôÒ¶¹‰¿ disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 26.734375# memory_percent‚95.8916187286377 network.sent‚ 2.330747e+06( disk./.usagePercent‚76.5578135316641 disk.dm-0.in‚0 disk.dm-0.out‚11612.69921875' disk.nvme0c0n1.out‚7432.72119140625' disk.nvme1c1n1.out‚5299.80419921875% disk.nvme1n1.out‚5299.80419921875" proc.memory.rssMB‚ 2936.5703125$ disk./.usageGB‚1175.353832244873* proc.memory.percent‚17.923402786254883 cpu‚25.87056146734074 proc.cpu.threads‚17% disk.nvme0n1.out‚7432.72119140625 network.recv‚345327! disk.dm-1.out‚1061.9931640625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0' proc.memory.availableMB‚ 673.1171875è†+žì‚(:“ ïôÒ—æ¿ gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.727' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.242333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò£Â9.í‚(:£ ŸïôÒˉ„¿# disk.nvme1c1n1.out‚ 5365.7734375 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 5365.7734375 disk.md0.out‚ 27.0859375 disk.dm-0.out‚ 11860.296875 disk.dm-1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚76.55365556755122 disk.md0.in‚0& disk.nvme0c0n1.out‚7615.2412109375 disk.nvme0n1.in‚0" proc.memory.rssMB‚ 2936.5703125 proc.cpu.threads‚17 network.sent‚ 2.394911e+06# disk./.usageGB‚1175.28999710083 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0# memory_percent‚95.8916425704956* proc.memory.percent‚17.923402786254883 cpu‚25.87376268326927 disk.nvme1c1n1.in‚0! disk.dm-1.out‚1062.1259765625 disk.rbd0.out‚0$ disk.nvme0n1.out‚7615.2412109375( proc.memory.availableMB‚ 673.11328125 network.recv‚349711+’‘î‚(:† ŸïôÒ¬·¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.861' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.287 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ4à_2ï- _wandbruntime‚1143  _runtime‚1143$‘ê±Èð _runtime‚1143.650350194 _step‚136 ' lr-AdamW/pg1‚0.00015871756449518479 ' lr-AdamW/pg2‚0.00015871756449518479 trainer/global_step‚1699 $ _timestamp‚1.784493987046952e+09ØõÂDÍñÇ ' lr-AdamW/pg2‚0.00015871756449518479 trainer/global_step‚1699 $ _timestamp‚1.784493987046952e+09 _runtime‚1143.650350194 _step‚136 ' lr-AdamW/pg1‚0.00015871756449518479ˆ*¹2ò- _wandbruntime‚1143  _runtime‚1143¨‡ËÜ'óÖO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9029517 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130206181932080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133159933916672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1400277118976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98692856510976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12883095 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1442333485568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844939870496068e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133159828217344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133159933916672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100135396772864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131602719006546 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1400365778944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17094270 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17084127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1400277118976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131759568137728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460471 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17094270 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883275 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133159828217344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100135189996544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131602615180760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1396433248680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚1699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460473 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460471 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98692950620672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206776320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1396520035106 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30544598 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 112666624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 _step‚137 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131759551098368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131759551098368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13460473 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3853578 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1442446152192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9029521 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844939870496068e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133159828217344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133159933916672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100135396772864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131602719006546 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1400365778944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17094270 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17084127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1400277118976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131759568137728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460471 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17094270 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883275 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133159828217344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100135189996544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131602615180760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1396433248680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚1699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460473 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460471 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98692950620672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0‰áîYÜ2õ- _wandbruntime‚1144  _runtime‚1144¹ƒÖˆößM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98749088348672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3855696 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131677689200752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130280489496832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 trainer/global_step‚1699 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468100 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1401151501824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131834717804032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17104104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12890269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3855873 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 _runtime‚1144.067902189 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_ez]train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1401151501824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13468102 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100192140472832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚181 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131834700764672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30562047 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17093947 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1397303529818 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131834700764672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1401062834688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4‘Zšê&÷äM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131834717804032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17104104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12890269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3855873 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 _runtime‚1144.067902189 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1401151501824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13468102 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100192140472832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚181 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131834700764672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30562047 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17093947 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1397303529818 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131834700764672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1401062834688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98749088348672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3855696 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131677689200752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130280489496832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 trainer/global_step‚1699 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468100 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1401151501824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Šc¢2ø- _wandbruntime‚1144  _runtime‚1144µI»ù 0loss/FRt2/assignment_loss‚2.9135233281528703 /loss/FRt2/detection_loss‚0.5820935390599009 1loss/SRqqt2/detection_loss‚1.4250692557658766 epoch‚4 1loss/FBt1/assignment_loss‚0.05860126839452436 1loss/FBt2/assignment_loss‚0.21845447543801744 /loss/FRt1/detection_loss‚1.3771889239461077 1loss/SRqqt1/detection_loss‚0.7233733828362086 &loss/total_loss‚13.421210756288172 trainer/global_step‚1699 2loss/SRqqt1/assignment_loss‚0.7794656899400676 /loss/FBt1/detection_loss‚1.6630882238161346 _runtime‚ 1144.06847453 0loss/FRt1/assignment_loss‚1.3333070814216097 2loss/SRqqt2/assignment_loss‚0.8788409136193857 /loss/FBt2/detection_loss‚1.4682046738974683 % _timestamp‚1.7844939874669466e+09 _step‚139;CÚ‚ ú† 1loss/FBt1/assignment_loss‚0.05860126839452436 1loss/FBt2/assignment_loss‚0.21845447543801744 /loss/FRt1/detection_loss‚1.3771889239461077 1loss/SRqqt1/detection_loss‚0.7233733828362086 &loss/total_loss‚13.421210756288172 trainer/global_step‚1699 2loss/SRqqt1/assignment_loss‚0.7794656899400676 /loss/FBt1/detection_loss‚1.6630882238161346 _runtime‚ 1144.06847453 0loss/FRt1/assignment_loss‚1.3333070814216097 2loss/SRqqt2/assignment_loss‚0.8788409136193857 /loss/FBt2/detection_loss‚1.4682046738974683 % _timestamp‚1.7844939874669466e+09 _step‚139 0loss/FRt2/assignment_loss‚2.9135233281528703 /loss/FRt2/detection_loss‚0.5820935390599009 1loss/SRqqt2/detection_loss‚1.4250692557658766 epoch‚4‹ÚÂN(5û‚(:ª ®ïôÒ°ì“¿ proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚7616.009765625# disk.nvme0n1.out‚7616.009765625 disk.nvme1n1.in‚0 disk.md0.out‚ 27.3203125 disk.dm-0.in‚0$ memory_percent‚95.89331150054932 network.sent‚ 2.410783e+06 network.recv‚353254 disk.dm-1.in‚0$ disk.nvme1n1.out‚5366.3076171875 cpu‚25.87623236961827) disk./.usagePercent‚76.55417636907987% disk./.usageGB‚1175.2979927062988! disk.dm-1.out‚1062.2822265625 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚5366.3076171875 disk.md0.in‚0( proc.memory.availableMB‚ 672.83984375 disk.rbd0.in‚0 disk.dm-0.out‚ 11860.921875" proc.memory.rssMB‚ 2936.5703125* proc.memory.percent‚17.923402786254883¬Rì‘ü‚(:† ®ïôÒ“¬È¿ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚216.585' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚72.195 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a¥Ø]9ý‚(:® ½ïôÒ¥ùü¾ disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚5368.447265625" proc.memory.rssMB‚ 2936.5703125* proc.memory.percent‚17.923402786254883 cpu‚25.874641233313426 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 27.6796875 disk.dm-0.in‚0 disk.nvme0n1.in‚0% disk.nvme1c1n1.out‚5368.447265625 network.recv‚357865 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7631.57958984375$ memory_percent‚95.89290618896484% proc.memory.availableMB‚ 672.90625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 network.sent‚ 2.46048e+06) disk./.usagePercent‚76.55418829583245% disk./.usageGB‚1175.2981758117676 disk.dm-1.in‚0! disk.dm-1.out‚1062.3447265625% disk.nvme0n1.out‚7631.57958984375# disk.dm-0.out‚11877.77392578125UÎ52þ‚(:’ ½ïôÒî׿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.884' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.294666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ Lè2ÿ- _wandbruntime‚1174  _runtime‚11748éuÇ€ Á _step‚140 & lr-AdamW/pg1‚0.0001585821422375263 & lr-AdamW/pg2‚0.0001585821422375263 trainer/global_step‚1749 % _timestamp‚1.7844940180809047e+09 _runtime‚1174.684303926å(tÌ Æ & lr-AdamW/pg1‚0.0001585821422375263 & lr-AdamW/pg2‚0.0001585821422375263 trainer/global_step‚1749 % _timestamp‚1.7844940180809047e+09 _runtime‚1174.684303926 _step‚140Œ lÖ2‚ - _wandbruntime‚1174  _runtime‚1174ãVþÉÞ'ƒ ØO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13841921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1482957843968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 trainer/global_step‚1749 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13841921 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17576979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17587122 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3962402 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101499742910976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9282321 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9282317 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136956767172096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135517051452928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135517051452928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1439715719168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136956661472768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7844940180814583e+09 _runtime‚1174.684363896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1439715719168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102982700754944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚194 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17587122 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13244723 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1439627059200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13841923 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135517034413568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1435758565914 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1483080996352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102982918017024 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136956767172096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31429045 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13841923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13244525 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962208 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136956661472768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7844940180814583e+09 _runtime‚1174.684363896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1439715719168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102982700754944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚194 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17587122 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13244723 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1439627059200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13841923 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135517034413568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1435758565914 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1483080996352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102982918017024 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136956767172096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31429045 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13841923 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13244525 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962208 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13841921 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17576979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17587122 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3962402<ÁfÞ2… - _wandbruntime‚1175  _runtime‚1175´L3åè&† âM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135592201119232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135592184079872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚1749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135592184079872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1440501442048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101555974748672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103039871083008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1175.078151352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137032702561280 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137032596854784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1440501442048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103039645439488 _step‚142 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13251779 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135431557827058 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135431454001160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17596956 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1440412774912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚195 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13849552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133995015766432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17586799 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3964601 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3964406 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1436455274088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31436349 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1483764800512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101555880638976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13849550 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137032702561280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13849552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13251978 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1483896334336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1436542060626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 % _timestamp‚1.7844940184761167e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1440412774912 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1436542060626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 % _timestamp‚1.7844940184761167e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1440412774912 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135592201119232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135592184079872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚1749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135592184079872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1440501442048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101555974748672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103039871083008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1175.078151352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137032702561280 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137032596854784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1440501442048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103039645439488 _step‚142 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13251779 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135431557827058 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135431454001160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17596956 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1440412774912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚195 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13849552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133995015766432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17586799 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3964601 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3964406 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1436455274088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31436349 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1483764800512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Ž¾5* 2ˆ - _wandbruntime‚1175  _runtime‚1175ýy@º‰ ý epoch‚4 0loss/FRt2/assignment_loss‚2.9220456227238447 /loss/FRt1/detection_loss‚1.3682455546008803 /loss/FRt2/detection_loss‚0.6133204059733544 /loss/SRqqt1/detection_loss‚0.78781142204591 &loss/total_loss‚13.963740794506965 % _timestamp‚1.7844940184774075e+09 _runtime‚1175.078935043 0loss/FRt1/assignment_loss‚1.3609328898442845 1loss/FBt2/assignment_loss‚0.14502781149791455 trainer/global_step‚1749 1loss/FBt1/assignment_loss‚0.14553726831208982 /loss/FBt1/detection_loss‚1.4834028122043414 .loss/FBt2/detection_loss‚2.089855718877742 _step‚143 2loss/SRqqt1/assignment_loss‚0.8068505140422361 0loss/SRqqt2/assignment_loss‚0.91694336720896 1loss/SRqqt2/detection_loss‚1.3237674071754062È¢Õ3Š ‚ 2loss/SRqqt1/assignment_loss‚0.8068505140422361 0loss/SRqqt2/assignment_loss‚0.91694336720896 1loss/SRqqt2/detection_loss‚1.3237674071754062 epoch‚4 0loss/FRt2/assignment_loss‚2.9220456227238447 /loss/FRt1/detection_loss‚1.3682455546008803 /loss/FRt2/detection_loss‚0.6133204059733544 /loss/SRqqt1/detection_loss‚0.78781142204591 &loss/total_loss‚13.963740794506965 % _timestamp‚1.7844940184774075e+09 _runtime‚1175.078935043 0loss/FRt1/assignment_loss‚1.3609328898442845 1loss/FBt2/assignment_loss‚0.14502781149791455 trainer/global_step‚1749 1loss/FBt1/assignment_loss‚0.14553726831208982 /loss/FBt1/detection_loss‚1.4834028122043414 .loss/FBt2/detection_loss‚2.089855718877742 _step‚143ƒ±Í²=‹ ‚(:² ÌïôÒØ‚ý¾$ memory_percent‚95.89321613311768( proc.memory.availableMB‚ 672.85546875 cpu‚25.871205084885762 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5369.080078125" proc.memory.rssMB‚ 2936.5859375* proc.memory.percent‚17.923498153686523# disk.dm-0.out‚11878.32470703125! disk.dm-1.out‚1062.4990234375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17 network.recv‚361091) disk./.usagePercent‚76.55419674394885% disk./.usageGB‚1175.2983055114746 disk.nvme1n1.in‚0# disk.nvme1n1.out‚5369.080078125 disk.md0.in‚0% disk.nvme0n1.out‚7632.40576171875 network.sent‚ 2.465234e+06 disk.md0.out‚ 28.0234375 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚7632.40576171875:yŸ÷Œ ‚(:ˆ ÌïôÒ³ïÞ¿ gpu.0.gpu‚84.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚116.325' gpu.0.enforcedPowerLimitWatts‚300.«^Ó"œ0 gpu.0.powerPercent‚38.775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18069831 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1474997096722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1523560713216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140753600427520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13606387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚137635251510640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18079974 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139274534768128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139274517728768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18079974 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13606209 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9535117 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1523685962752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139110248607362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1478976999424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139110144781576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140753600427520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104306723420672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4071092 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105830409383424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1474910310296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1478976999424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1479065659392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14223373 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4071266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14223371 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140753494728192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104306629310976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32303347 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137635234471280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _step‚145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18069831 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1205.118829687 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14223373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1479065659392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚178 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9535121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140753494728192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139274517728768 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4071266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14223371 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140753494728192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104306629310976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32303347 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137635234471280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _step‚145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18069831 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1205.118829687 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14223373 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1479065659392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚178 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9535121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140753494728192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139274517728768 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18069831 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1474997096722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1523560713216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140753600427520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13606387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚137635251510640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18079974 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139274534768128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139274517728768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18079974 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13606209 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9535117 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1523685962752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139110248607362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1478976999424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130‘ÇA[2• - _wandbruntime‚1205  _runtime‚1205EvÑè&– âM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137709542036032 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1479762715136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137709524996672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1475693804896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13613552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1524373854720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139185322627466 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139349684434432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1524490708480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1475780591434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104362767038976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 trainer/global_step‚1799 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18089808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4073375 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231002 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140829535816704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32320810 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1479851382272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1479851382272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140829430110208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139349667395072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1479762715136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9540173 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140829535816704 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚146 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105887140893696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104362861148672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844940489119194e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18079651 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073203 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13613376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18089808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139349684434432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9540177 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13613552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1524373854720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139185322627466 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139349684434432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1524490708480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1475780591434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104362767038976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 trainer/global_step‚1799 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18089808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4073375 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231002 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140829535816704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32320810 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1479851382272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1479851382272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140829430110208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139349667395072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1479762715136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9540173 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140829535816704 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚146 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105887140893696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104362861148672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844940489119194e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18079651 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073203 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13613376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18089808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139349684434432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9540177 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18572826 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 % _timestamp‚1.7844940793022316e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18572826 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9787921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13965790 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1514148841104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18562683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1518326939648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144550327983616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143032018083328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108677781412864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1564040051200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚193 trainer/global_step‚1849 _step‚149 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108677555762176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1514235627530 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14604823 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚197 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 _runtime‚ 1235.90727671 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18572826 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 % _timestamp‚1.7844940793022316e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18572826 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9787921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13965790 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1514148841104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18562683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1518326939648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144550327983616 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143032018083328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108677781412864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1564040051200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ ”é ;~269300736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚193 trainer/global_step‚1849 _step‚149 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108677555762176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1514235627530 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14604823 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚197 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 _runtime‚ 1235.90727671 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33184953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14612452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1236.307309384 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1514932335704 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1519112655360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144626369072128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚150 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13972863 trainer/global_step‚1849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4180059 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18572503 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1519201322496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4179890 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1519201322496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142938983601976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1519112655360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141424068305632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279˱ ýí&§ çM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1519201322496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142938983601976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1519112655360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141424068305632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33184953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14612452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1236.307309384 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1514932335704 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1519112655360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144626369072128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚150 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13972863 trainer/global_step‚1849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4180059 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18572503 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1519201322496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4179890 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760–ÓE/12¨ - _wandbruntime‚1236  _runtime‚1236XuΩ  2loss/SRqqt2/assignment_loss‚0.8603354860422807 0loss/FBt2/assignment_loss‚0.2468121885091727 /loss/FRt1/detection_loss‚1.3664090583028008 1loss/SRqqt1/detection_loss‚0.7895347559935251 _runtime‚1236.307779825 /loss/FBt2/detection_loss‚1.3399789074384978 &loss/total_loss‚12.774358731363607 epoch‚4 0loss/FRt2/assignment_loss‚2.9704667504600417 /loss/FRt2/detection_loss‚0.5596035252931584 1loss/SRqqt2/detection_loss‚1.3751307230656868 trainer/global_step‚1849 % _timestamp‚1.7844940797062285e+09 _step‚151 0loss/FRt1/assignment_loss‚1.3345673986356266 2loss/SRqqt1/assignment_loss‚0.7630491990831749 1loss/FBt1/assignment_loss‚0.16452965199460745 /loss/FBt1/detection_loss‚1.00394108654503383ñb ª † epoch‚4 0loss/FRt2/assignment_loss‚2.9704667504600417 /loss/FRt2/detection_loss‚0.5596035252931584 1loss/SRqqt2/detection_loss‚1.3751307230656868 trainer/global_step‚1849 % _timestamp‚1.7844940797062285e+09 _step‚151 0loss/FRt1/assignment_loss‚1.3345673986356266 2loss/SRqqt1/assignment_loss‚0.7630491990831749 1loss/FBt1/assignment_loss‚0.16452965199460745 /loss/FBt1/detection_loss‚1.0039410865450338 2loss/SRqqt2/assignment_loss‚0.8603354860422807 0loss/FBt2/assignment_loss‚0.2468121885091727 /loss/FRt1/detection_loss‚1.3664090583028008 1loss/SRqqt1/detection_loss‚0.7895347559935251 _runtime‚1236.307779825 /loss/FBt2/detection_loss‚1.3399789074384978 &loss/total_loss‚12.774358731363607—¹²&X'« ‚(:œ ˆðôÒù¡€¿) disk./.usagePercent‚76.57071678709507 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚5424.568359375 disk.dm-1.out‚ 1063.01171875" proc.memory.rssMB‚ 2936.5859375$ disk./.usageGB‚1175.551929473877 disk.nvme0n1.in‚0 disk.md0.out‚ 29.765625 disk.dm-0.in‚0 cpu‚25.85827816984777 proc.cpu.threads‚17 disk.md0.in‚0 network.recv‚375210% disk.nvme1c1n1.out‚5424.568359375 disk.nvme1n1.in‚0! disk.dm-0.out‚12145.232421875 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚95.89095115661621 disk.nvme0c0n1.out‚ 7848.0625 disk.nvme0n1.out‚ 7848.0625 disk.rbd0.in‚0' proc.memory.availableMB‚ 673.2265625* proc.memory.percent‚17.923498153686523 network.sent‚ 2.611542e+06gMÌž¬ ‚(:“ ˆðôҦ߯¿ gpu.0.gpu‚53.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚195.585' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.19500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àzy6­ ‚(:« —ðôÒ’烿 network.sent‚ 2.660989e+06 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚7863.68408203125% proc.memory.availableMB‚ 673.21875 proc.cpu.threads‚17# disk.dm-0.out‚12161.67431640625 disk.dm-1.out‚ 1063.05859375' disk.nvme0c0n1.out‚7863.68408203125$ memory_percent‚95.89099884033203 network.recv‚379569) disk./.usagePercent‚76.57072324741937# disk.nvme1n1.out‚5425.880859375 disk.md0.out‚29.96875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 cpu‚25.855235843659948% disk./.usageGB‚1175.5520286560059 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5425.880859375 disk.dm-0.in‚0" proc.memory.rssMB‚ 2936.5859375* proc.memory.percent‚17.923498153686523ÿ%R‘® ‚(:† —ðôÒøíÇ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.012' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚74.004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜üqÉ2¯ - _wandbruntime‚1267  _runtime‚1267ç``xÉ° à ' lr-AdamW/pg2‚0.00015814571552079702 trainer/global_step‚1899 % _timestamp‚1.7844941108200054e+09 _runtime‚1267.421844773 _step‚152 ' lr-AdamW/pg1‚0.00015814571552079702—ÌUα È ' lr-AdamW/pg1‚0.00015814571552079702 ' lr-AdamW/pg2‚0.00015814571552079702 trainer/global_step‚1899 % _timestamp‚1.7844941108200054e+09 _runtime‚1267.421844773 _step‚152˜gÔ‚/2² - _wandbruntime‚1267  _runtime‚1267>dža?³ ÚO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14325380 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1553474158338 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1557765539840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1557676879872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146789501398528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19055535 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10040717 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 trainer/global_step‚1899 _runtime‚1267.426248561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146789484359168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4284659 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19055535 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14986271 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109920402110976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146789484359168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14325209 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145064304049840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111524957395456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148347161239040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1604680534016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19065678 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148347266938368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148347161239040 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844941108209262e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34051951 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4284492 _step‚153 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148347161239040 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844941108209262e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34051951 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4284492 _step‚153 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146789501398528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19055535 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10040717 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 trainer/global_step‚1899 _runtime‚1267.426248561 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146789484359168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4284659 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19055535 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14986271 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109920402110976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146789484359168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14325209 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145064304049840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111524957395456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148347161239040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1604680534016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19065678 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148347266938368™œè52µ - _wandbruntime‚1267  _runtime‚1267Fµ3è&¶ âM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148423096621056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14332557 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1605496822272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4286780 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14993900 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146864634025472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111582130770944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14332379 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148423202327552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146864651064832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146692748402384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4286606 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1558551262720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19065355 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145138594575232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34059255 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109976539838976 _runtime‚ 1267.82453164 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚1899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1558551262720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1558462595584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14993900 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146692852228282 _step‚154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111581903030272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚1899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1558551262720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1558462595584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14993900 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146692852228282 _step‚154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111581903030272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4286780 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14993900 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146864634025472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \HDeviceStatsMonitor.on_train_batch_endvìÇ°/inactive_split_bytes.all.allocated‚111582130770944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14332379 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148423202327552 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146864651064832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146692748402384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4286606 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1558551262720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19065355 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812šá¤K2¸ - _wandbruntime‚1267  _runtime‚1267Äë°¹ ÿ $ _timestamp‚1.784494111223497e+09 0loss/FRt2/assignment_loss‚2.7170513499511313 /loss/FBt1/detection_loss‚1.1419577642147087 &loss/total_loss‚16.151566793925745 epoch‚4 _step‚155 2loss/SRqqt1/assignment_loss‚0.7572714508450938 0loss/FBt2/assignment_loss‚0.2416973970365834 /loss/FRt1/detection_loss‚1.3740058451544934 2loss/SRqqt2/assignment_loss‚0.8370320737941942 1loss/FBt1/assignment_loss‚0.12938015509784792 1loss/SRqqt1/detection_loss‚0.7351280400675999 1loss/SRqqt2/detection_loss‚1.3243007549378756 trainer/global_step‚1899 _runtime‚1267.824996461 0loss/FRt1/assignment_loss‚1.3189143914108474 /loss/FRt2/detection_loss‚0.6718162684507797 .loss/FBt2/detection_loss‚4.903011302964593{;,Ø º „ 0loss/FRt2/assignment_loss‚2.7170513499511313 /loss/FBt1/detection_loss‚1.1419577642147087 &loss/total_loss‚16.151566793925745 epoch‚4 _step‚155 2loss/SRqqt1/assignment_loss‚0.7572714508450938 0loss/FBt2/assignment_loss‚0.2416973970365834 /loss/FRt1/detection_loss‚1.3740058451544934 2loss/SRqqt2/assignment_loss‚0.8370320737941942 1loss/FBt1/assignment_loss‚0.12938015509784792 1loss/SRqqt1/detection_loss‚0.7351280400675999 1loss/SRqqt2/detection_loss‚1.3243007549378756 trainer/global_step‚1899 _runtime‚1267.824996461 0loss/FRt1/assignment_loss‚1.3189143914108474 /loss/FRt2/detection_loss‚0.6718162684507797 .loss/FBt2/detection_loss‚4.903011302964593 $ _timestamp‚1.784494111223497e+09›!Yž» ‚(:“ ¦ðôÒŒ‰†¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚219.029' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.00966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î?C{5¼ ‚(:ª ¦ðôÒê•¿& disk.nvme0c0n1.out‚7865.0380859375 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.92922019958496 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚7865.0380859375$ disk.nvme1c1n1.out‚ 5426.51953125& proc.memory.availableMB‚ 607.484375" proc.memory.rssMB‚ 2937.5234375 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 5426.51953125) disk./.usagePercent‚76.57078014796807% disk./.usageGB‚1175.5529022216797 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚0" disk.dm-1.out‚1063.23388671875 disk.md0.out‚30.28125 disk.dm-0.in‚0# disk.dm-0.out‚12162.81494140625$ memory_percent‚96.29220962524414 cpu‚25.851677663100684 proc.cpu.threads‚17 network.sent‚ 2.678686e+06 network.recv‚383230¨ˆÓqB½ ‚(:· µðôÒ¨Öݾ$ memory_percent‚96.29218578338623' disk.nvme0c0n1.out‚7869.31103515625 disk.nvme1c1n1.in‚0" disk.dm-1.out‚1063.29638671875( proc.memory.availableMB‚ 607.48828125) proc.memory.percent‚17.92922019958496 cpu‚25.848417060148762 proc.cpu.threads‚17% disk./.usageGB‚1175.3151321411133' disk.nvme1c1n1.out‚5427.53662109375 disk.nvme1n1.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚76.55529276281474 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 30.68359375# disk.dm-0.out‚12167.18798828125" proc.memory.rssMB‚ 2937.5234375% disk.nvme1n1.out‚5427.53662109375 network.sent‚ 2.728236e+06 network.recv‚387836 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚7869.31103515625 disk.dm-0.in‚0ÛxŸ¾ ‚(:” µðôÒô—£¿ gpu.0.gpu‚66.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.184' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.394666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Þcv2¿ - _wandbruntime‚1298  _runtime‚1298&]F3ÈÀ  ' lr-AdamW/pg1‚0.00015799021777944988 ' lr-AdamW/pg2‚0.00015799021777944988 trainer/global_step‚1949 $ _timestamp‚1.784494142252679e+09 _runtime‚1298.855294797 _step‚156„y£ÍÁ Ç ' lr-AdamW/pg1‚0.00015799021777944988 ' lr-AdamW/pg2‚0.00015799021777944988 trainer/global_step‚1949 $ _timestamp‚1.784494142252679e+09 _runtime‚1298.855294797 _step‚156œÇ3òu2 - _wandbruntime‚1298  _runtime‚12981£Äàß'à ÙO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150546984713728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1597026820096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4389867 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114372371947008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1597026820096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15367721 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152143994494464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19548387 _step‚157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150371543008586 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112727382620672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34926253 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148778830319440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15367723 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10293521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1592712689146 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148778813280080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 % _timestamp‚1.7844941422533116e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152143994494464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19558530 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚177 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112727288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1645083436032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1645214977024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14683565 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150546967674368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760…žä'Ä ÞO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 % _timestamp‚1.7844941422533116e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152143994494464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19558530 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚177 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112727288510976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1645083436032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1645214977024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14683565 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150546967674368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150546984713728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1597026820096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4389867 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114372371947008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1597026820096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15367721 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152143994494464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19548387 _step‚157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150371543008586 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112727382620672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34926253 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148778830319440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15367723 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10293521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1592712689146 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148778813280080ìùø}2Å - _wandbruntime‚1299  _runtime‚12997Œn ýÆ ãM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148853120844832 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1593496183858 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1597812535808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14690415 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150622117340672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4392018 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_b¥Õ@~ìytes.small_pool.allocated‚ 272629760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152219929876480 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15375350 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114429541382144 % _timestamp‚1.7844941426532493e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34943716 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19568364 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150446617028690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19558207 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4391842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1646021033472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175âoxÒî&Ç èM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112783426238976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150622134380032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15375352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14690595 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150622134380032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1593409397320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _runtime‚ 1299.25533972 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4392018 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152219929876480 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15375350 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114429541382144 % _timestamp‚1.7844941426532493e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34943716 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19568364 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150446617028690 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19558207 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4391842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1646021033472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148853120844832 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1593496183858 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1597812535808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14690415 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150622117340672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4496486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1636465420288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154304450989568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15749173 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154125307808994 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10546317 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152493356589040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20051382 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155940827749888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152493339549680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15749171 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20041239 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115534174910976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636376760320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15749173 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1685560890368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20051382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15042982 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15749171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117219959354880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1636465420288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1631864433528 trainer/global_step‚1999 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10546321 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155940933449216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1685690334208 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155940827749888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152493339549680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15749171 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20041239 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115534174910976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636376760320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15749173 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1685560890368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20051382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15042982 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15749171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.akîÔÞllocated‚117219959354880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1636465420288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1631864433528 trainer/global_step‚1999 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10546321 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155940933449216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1685690334208 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4496486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1636465420288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154304450989568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15749173¡·žI2Õ - _wandbruntime‚1330  _runtime‚1330_n#é&Ö ãM YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚1999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1637251143168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4498533 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20051059 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117276906129408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1686499380736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15756800 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115590406748672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1686365749760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20061216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15050085 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154200278003200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚ 1330.25966255 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154379617695232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152567647114432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20051059 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1632734714666 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15756802 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156016868838400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15756802 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115590312638976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15049906 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1637162476032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736DÄqÕî&× èM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154379617695232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152567647114432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20051059 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1632734714666 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15756802 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156016868838400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15756802 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115590312638976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15049906 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1637162476032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚1999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1637251143168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4498533 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20051059 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117276906129408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1686499380736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15756800 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115590406748672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1686365749760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20061216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15050085 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154200278003200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚ 1330.25966255 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175¢5jX72Ø - _wandbruntime‚1330  _runtime‚1330ž?lÒÙ ý 2loss/SRqqt1/assignment_loss‚0.7292885599015121 .loss/FRt1/detection_loss‚1.405448987705319 /loss/FRt2/detection_loss‚0.6470096608768665 trainer/global_step‚1999 /loss/FRt1/assignment_loss‚1.285450848096201 2loss/SRqqt2/assignment_loss‚0.9365073437000296 1loss/FBt1/assignment_loss‚0.07920853186246085 0loss/FBt2/assignment_loss‚0.1423854501054843 1loss/SRqqt2/detection_loss‚1.3561861933845603 /loss/FBt1/detection_loss‚0.8915760196252753 /loss/FBt2/detection_loss‚1.2278141227031873 %loss/total_loss‚12.30491731084325 epoch‚4 _runtime‚1330.260101361 1loss/SRqqt1/detection_loss‚0.8149671812874585 $ _timestamp‚1.784494173658588e+09 _step‚163 0loss/FRt2/assignment_loss‚2.7890744115948944'ÈV(Ú ‚ 0loss/FRt2/assignment_loss‚2.7890744115948944 2loss/SRqqt1/assignment_loss‚0.7292885599015121 .loss/FRt1/detection_loss‚1.405448987705319 /loss/FRt2/detection_loss‚0.6470096608768665 trainer/global_step‚1999 /loss/FRt1/assignment_loss‚1.285450848096201 2loss/SRqqt2/assignment_loss‚0.9365073437000296 1loss/FBt1/assignment_loss‚0.07920853186246085 0loss/FBt2/assignment_loss‚0.1423854501054843 1loss/SRqqt2/detection_loss‚1.3561861933845603 /loss/FBt1/detection_loss‚0.8915760196252753 /loss/FBt2/detection_loss‚1.2278141227031873 %loss/total_loss‚12.30491731084325 epoch‚4 _runtime‚1330.260101361 1loss/SRqqt1/detection_loss‚0.8149671812874585 $ _timestamp‚1.784494173658588e+09 _step‚163£þ­„7Û ‚(:¬ âðôÒ®¿" disk.nvme0n1.out‚ 7879.72265625' disk.nvme1c1n1.out‚5429.67236328125 disk.md0.out‚ 31.91796875 disk.rbd0.out‚0 proc.memory.rssMB‚ 2938.84375* proc.memory.percent‚17.937278747558594 cpu‚25.85029792479008% disk./.usageGB‚1175.3146133422852 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚12176.62109375 disk.dm-1.in‚0 disk.rbd0.in‚0 network.sent‚ 2.800048e+06 network.recv‚398903 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0$ disk.nvme0c0n1.out‚ 7879.72265625 proc.cpu.threads‚17) disk./.usagePercent‚76.55525897034914% disk.nvme1n1.out‚5429.67236328125" disk.dm-1.out‚1063.73486328125 disk.nvme0c0n1.in‚0$ memory_percent‚96.30162715911865( proc.memory.availableMB‚ 605.94140625>ãœÜ ‚(:‘ âðôÒÄßñ¾ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.458' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.15266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fÿõø2Ý - _wandbruntime‚1343  _runtime‚1343ÔrcŠÞ ë{ particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9313080310821533 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918867945671082 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897385835647583 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.704211950302124 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8080878257751465 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9927461743354797 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2045735865831375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7017677426338196 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7844941870896354e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7127642631530762 *jet/accuracy_1_of_4‚0.9566217064857483 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3407677710056305 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5300554633140564 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9963369965553284 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚12.274065933138477 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.28735366463661194 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _runtime‚1343.691939893 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6170074939727783 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1414707601070404 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2177506685256958 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1268801987171173 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5677054524421692 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.972748875617981 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6283258199691772 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6734945178031921 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.589349091053009 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.15526415407657623 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2676970958709717 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9822134971618652 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5757079720497131 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5209695100784302 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12447260320186615 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15305718779563904 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5757079720497131 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚1999 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 +jet/accuracy_3_of_3‚0.04951276630163193 & Purity/2FX6úMg/Rt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5941760540008545 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3402777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5454745292663574 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2539469301700592 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4615384638309479 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4669117331504822 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5663014650344849 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9783107042312622 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9280446171760559 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_3‚0.903774082660675 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7115384340286255 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7709372639656067 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_3‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19693267345428467 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.677998423576355 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9953755140304565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20808476209640503 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2870364785194397 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3642849624156952 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5340345501899719 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9419248700141907 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20798441767692566 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6213214993476868 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2666153311729431 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.30244457721710205 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333730697632 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9814162254333496 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5285951495170593 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.36736682057380676 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2090999037027359 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5625130534172058 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16539256274700165 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333432674408 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6202674508094788 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20725704729557037 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9927461743354797 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7895124554634094 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.563295841217041 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9772507548332214 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2184569239616394 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8136696815490723 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8838496208190918 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02126315049827099 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2790769636631012 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 +particle/sensitivity‚0.5844060778617859 epoch‚4 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5923702120780945 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6853081583976746 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.040788549929857254 *jet/accuracy_1_of_2‚0.7699132561683655 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4615384638309479 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03466327115893364 +particle/specificity‚0.9023043513298035 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5448592901229858 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6184661984443665 +jet/accuracy_2_of_2‚0.11581192910671234 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21629442274570465 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19755515456199646 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8055555820465088 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9754924774169922 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9698379039764404 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5893734693527222 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15802156925201416 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7912130951881409 .particle/accuracy_3_of_3‚0.997555673122406 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19105374813079834 'particle/f_score‚0.6019680500030518 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_2_of_3‚0.46616634726524353 _step‚164 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5401197075843811 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19665786623954773 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.532655656337738 (particle/accuracy‚0.8340346813201904 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.06836063414812088 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.94536292552948 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1831912100315094 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20725704729557037 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9939349293708801 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6702849268913269 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.31481480598449707 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967347383499146 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7143664956092834 ,jet/accuracy_4_of_4‚0.002801120514050126 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6133721470832825 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15388917922973633 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.359375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7615073323249817 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20607951283454895 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5943660140037537 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5735152363777161 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09833333641290665 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923804402351379 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1414707601070404 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7281123995780945 particle/accuracy_3_of_5‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03999244049191475 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.016282856464385986 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2566301226615906 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5225168466567993 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03071574866771698 7validation_average_jet_accuracy‚0.34405872225761414 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3370310366153717 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21784912049770355 *jet/accuracy_2_of_4‚0.7495604157447815 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.31481480598449707 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4937500059604645 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1996646374464035 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2770702838897705 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5750858783721924 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9856756329536438 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9934985637664795 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1576301008462906 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7281123995780945 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9563064575195312 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18516623973846436 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.664320707321167 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5797430276870728 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4650823175907135 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 jet/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5357241630554199 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.171312153339386 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8692687153816223 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7744519114494324 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6238388419151306 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8321959376335144 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15923146903514862 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0§bÞö=ß ð{ 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1996646374464035 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2770702838897705 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5750858783721924 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9856756329536438 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9934985637664795 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1576301008462906 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7281123995780945 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9563064575195312 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18516623973846436 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.664320707321167 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5797430276870728 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4650823175907135 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 jet/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5357241630554199 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.171312153339386 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7175925374031067 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8692687153816223 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7744519114494324 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6238388419151306 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8321959376335144 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15923146903514862 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9313080310821533 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9918867945671082 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897385835647583 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.704211950302124 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8080878257751465 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9927461743354797 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2045735865831375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7017677426338196 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7844941870896354e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7127642631530762 *jet/accuracy_1_of_4‚0.9566217064857483 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3407677710056305 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5300554633140564 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9963369965553284 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚12.274065933138477 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.28735366463661194 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _runtime‚1343.691939893 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6170074939727783 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1414707601070404 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2177506685256958 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1268801987171173 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5677054524421692 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.972748875617981 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6283258199691772 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6734945178031921 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.589349091053009 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.15526415407657623 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2676970958709717 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9822134971618652 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5757079720497131 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5209695100784302 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12447260320186615 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15305718779563904 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5757079720497131 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚1999 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 +jet/accuracy_3_of_3‚0.04951276630163193 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5941760540008545 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3402777910232544 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5454745292663574 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2539469301700592 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4615384638309479 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4669117331504822 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5663014650344849 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9783107042312622 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9280446171760559 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_3‚0.903774082660675 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7115384340286255 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7709372639656067 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_3‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19693267345428467 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.677998423576355 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9953755140304565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20808476209640503 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2870364785194397 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3642849624156952 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5340345501899719 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7175925374031067 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9419248700141907 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20798441767692566 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6213214993476868 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2666153311729431 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.30244457721710205 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5833333730697632 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9814162254333496 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5285951495170593 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.36736682057380676 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2090999037027359 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5625130534172058 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16539256274700165 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333432674408 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6202674508094788 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20725704729557037 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9927461743354797 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7895124554634094 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.563295841217041 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9772507548332214 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2184569239616394 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8136696815490723 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8838496208190918 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02126315049827099 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2790769636631012 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9967682361602783 +particle/sensitivity‚0.5844060778617859 epoch‚4 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5923702120780945 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6853081583976746 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.040788549929857254 *jet/accuracy_1_of_2‚0.7699132561683655 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4615384638309479 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03466327115893364 +particle/specificity‚0.9023043513298035 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5448592901229858 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6184661984443665 +jet/accuracy_2_of_2‚0.11581192910671234 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21629442274570465 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19755515456199646 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8055555820465088 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9754924774169922 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9698379039764404 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5893734693527222 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15802156925201416 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7912130951881409 .particle/accuracy_3_of_3‚0.997555673122406 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19105374813079834 'particle/f_score‚0.6019680500030518 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_2_of_3‚0.46616634726524353 _step‚164 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5401197075843811 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19665786623954773 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.532655656337738 (particle/accuracy‚0.8340346813201904 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.06836063414812088 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.94536292552948 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1831912100315094 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20725704729557037 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9939349293708801 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6702849268913269 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.31481480598449707 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967347383499146 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7143664956092834 ,jet/accuracy_4_of_4‚0.002801120514050126 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6133721470832825 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15388917922973633 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.359375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7615073323249817 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20607951283454895 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5943660140037537 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5735152363777161 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09833333641290665 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923804402351379 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1414707601070404 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7281123995780945 particle/accuracy_3_of_5‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03999244049191475 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.016282856464385986 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2566301226615906 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5225168466567993 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03071574866771698 7validation_average_jet_accuracy‚0.34405872225761414 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3370310366153717 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21784912049770355 *jet/accuracy_2_of_4‚0.7495604157447815 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.31481480598449707 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4937500059604645¤½ä0Aà ‚(:¶ ñðôÒÑý˜¿ disk.nvme0n1.in‚0% disk.nvme0n1.out‚7916.56884765625 disk.md0.out‚ 32.13671875 disk.rbd0.out‚0( proc.memory.availableMB‚ 593.07421875 network.recv‚403314 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0# memory_percent‚96.3801622390747# proc.memory.rssMB‚ 2952.05078125 cpu‚25.867897866178943' disk.nvme0c0n1.out‚7916.56884765625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5496.14990234375 disk.md0.in‚0# disk.dm-0.out‚12215.43896484375" disk.dm-1.out‚1127.75439453125 disk.rbd0.in‚0' disk./.usagePercent‚76.501993099499% disk.nvme1n1.out‚5496.14990234375* proc.memory.percent‚18.017888069152832 proc.cpu.threads‚17 network.sent‚ 2.876516e+06% disk./.usageGB‚1174.4968490600586ò©žá ‚(:“ ñðôÒÒíÇ¿ gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚222.454' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.15133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÚ®5â ‚(:ª €ñôÒìã¾$ disk.nvme1n1.out‚5618.8544921875 disk.dm-1.in‚0 disk.dm-1.out‚ 1191.96484375 disk.rbd0.in‚0$ memory_percent‚96.38171195983887 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚0' proc.memory.availableMB‚ 592.8203125* proc.memory.percent‚18.017888069152832 network.recv‚407195 disk.md0.in‚0 disk.md0.out‚32.46875 cpu‚25.86481882282264 proc.cpu.threads‚17$ disk.nvme0n1.out‚8106.2080078125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚5618.8544921875 disk.dm-0.in‚0! disk.dm-0.out‚12462.849609375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚8106.2080078125# proc.memory.rssMB‚ 2952.05078125 network.sent‚ 2.919695e+06) disk./.usagePercent‚76.51748992666478$ disk./.usageGB‚1174.734764099121DBÿžã ‚(:“ €ñôÒ–ÑÔ¾ gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.959' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.31966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=è‹2ä - _wandbruntime‚1374  _runtime‚1374c!˜Åå ¿ & lr-AdamW/pg2‚0.0001576642484839605 trainer/global_step‚2049 # _timestamp‚1.78449421836287e+09 _runtime‚1374.964684963 _step‚165 & lr-AdamW/pg1‚0.0001576642484839605ä÷ZÍÊæ Ä _step‚165 & lr-AdamW/pg1‚0.0001576642484839605 & lr-AdamW/pg2‚0.0001576642484839605 trainer/global_step‚2049 # _timestamp‚1.78449421836287e+09 _runtime‚1374.964684963¥éJqÁ2ç - _wandbruntime‚1374  _runtime‚1374"{®±è ÚO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20626741 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20616598 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4650228 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119144520625664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 112666624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160749683377664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16252421 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159058438640128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158867954904160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160749683377664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157181332855472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4650408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206776320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36869019 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159058438640128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16252423 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160749577678336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20616598 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1686605009328 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10907372 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160749577678336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157181349894832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1691156077568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1691156077568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36879164 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚166 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15557784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15557600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159058421600768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16252423 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120886150602240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10907376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1741724086272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2049 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120886357378560¾Øfå'é ßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20616598 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1686605009328 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10907372 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160749577678336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157181349894832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1691156077568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1691156077568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36879164 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚166 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15557784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15557600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159058421600768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16252423 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120886150602240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10907376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1741724086272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2049 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120886357378560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20626741 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20616598 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4650228 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119144520625664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 112666624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160749683377664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16252421 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159058438640128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158867954904160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160749683377664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157181332855472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4650408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206776320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36869019 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159058438640128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16252423 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160749577678336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486¦»µ]­2ê - _wandbruntime‚1375  _runtime‚1375¥°­é&ë ãM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10912432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119200564243968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15564818 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20636575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10912428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119200658353664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260052 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1691941793280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215157760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1687301717502 trainer/global_step‚2049 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1692030460416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158943028924264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36896627 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260050 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158942925098366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20636575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4652566 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1687388504040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120943096359936 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157255640420224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1691941793280 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4652390 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚1375.368436665 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159133588306432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159133571267072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260052 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159133571267072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160825513060352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 $ _timestamp‚1.784494218766339e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16260050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160825618766848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36886468 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120943311517696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 _step‚167 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160825618766848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20626418 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1742532115968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20626418 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1742653164032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36886468 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15564998 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812'–t­®ì èM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159133571267072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260052 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159133571267072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160825513060352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 $ _timestamp‚1.784494218766339e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16260050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160825618766848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36886468 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120943311517696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 _step‚167 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160825618766848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20626418 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1742532115968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20626418 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1742653164032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36886468 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15564998 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10912432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119200564243968 KEDeviceStatsMoåhâ”@nitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15564818 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20636575 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10912428 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119200658353664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16260052 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1691941793280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215157760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1687301717502 trainer/global_step‚2049 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1692030460416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158943028924264 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36896627 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260050 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158942925098366 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20636575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4652566 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1687388504040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120943096359936 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157255640420224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1691941793280 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4652390 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚1375.368436665 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159133588306432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F;2÷ - _wandbruntime‚1406  _runtime‚1406³_®à'ø ÚO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164546410933760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1725756753710 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16633873 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1725843540136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21119593 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21109450 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123733612781056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 trainer/global_step‚2099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162815904915968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162815921955328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164546516633088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1730506017792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162621615878782 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164546410933760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 _runtime‚1406.408433123 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1730594677760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 % _timestamp‚1.7844942498067086e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15918989 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160895859125072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15919172 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121951312915968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1782299865088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 _step‚170 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16633873 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1782299865088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 _step‚170 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16633873 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164546410933760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1725756753710 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16633873 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1725843540136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21119593 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21109450 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123733612781056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 trainer/global_step‚2099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162815904915968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162815921955328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164546516633088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1730506017792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162621615878782 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164546410933760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 _runtime‚1406.408433123 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1730594677760ªìMrE2ú - _wandbruntime‚1406  _runtime‚1406É5akû ãM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11165232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162696793724672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37760770 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1783109082624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1726627034848 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160970166689824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21129427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 E?DeviceStatsMonitor.on_train_batch_end/Ëa¨~ oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164622452022272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16641502 _runtime‚1406.810112642 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37760770 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15926386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162891071621632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123790783272960 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15926204 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16641500 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162891071621632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123790559726592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21129427 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1731291733504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1731291733504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160970149650464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _step‚171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16641502 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1731380400640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4761154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162891054582272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164622452022272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 $ _timestamp‚1.784494250208089e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122007544753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1731380400640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16641500 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11165228 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162696689898774 ¾wî&ü èM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164622452022272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 $ _timestamp‚1.784494250208089e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122007544753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1731380400640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16641500 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11165228 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162696689898774 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11165232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162696793724672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37760770 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1783109082624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1726627034848 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160970166689824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21129427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164622452022272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16641502 _runtime‚1406.810112642 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37760770 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15926386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162891071621632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123790783272960 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15926204 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16641500 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162891071621632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123790559726592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21129427 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1731291733504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1731291733504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160970149650464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _step‚171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16641502 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1731380400640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4761154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162891054582272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175«Œ_2ý - _wandbruntime‚1406  _runtime‚1406NôÖ½þ  /loss/FBt2/detection_loss‚1.3624584586148694 2loss/SRqqt2/assignment_loss‚0.9393940826395418 1loss/FBt1/assignment_loss‚0.08166657272703404 /loss/FBt1/detection_loss‚1.1063480991870036 epoch‚5 % _timestamp‚1.7844942502090364e+09 _step‚172 0loss/FRt2/assignment_loss‚2.8242388747133615 trainer/global_step‚2099 /loss/FRt1/detection_loss‚1.5294459803437348 1loss/SRqqt2/detection_loss‚1.4877283690898828 &loss/total_loss‚12.846044841700207 _runtime‚1406.810526932 0loss/FRt1/assignment_loss‚1.2443935175736744 1loss/SRqqt1/assignment_loss‚0.685436873490575 1loss/FBt2/assignment_loss‚0.16337497539951293 /loss/FRt2/detection_loss‚0.6131140816943724 1loss/SRqqt1/detection_loss‚0.8084449562266438hrh ÿ † 1loss/SRqqt1/assignment_loss‚0.685436873490575 1loss/FBt2/assignment_loss‚0.16337497539951293 /loss/FRt2/detection_loss‚0.6131140816943724 1loss/SRqqt1/detection_loss‚0.8084449562266438 /loss/FBt2/detection_loss‚1.3624584586148694 2loss/SRqqt2/assignment_loss‚0.9393940826395418 1loss/FBt1/assignment_loss‚0.08166657272703404 /loss/FBt1/detection_loss‚1.1063480991870036 epoch‚5 % _timestamp‚1.7844942502090364e+09 _step‚172 0loss/FRt2/assignment_loss‚2.8242388747133615 trainer/global_step‚2099 /loss/FRt1/detection_loss‚1.5294459803437348 1loss/SRqqt2/detection_loss‚1.4877283690898828 &loss/total_loss‚12.846044841700207 _runtime‚1406.810526932 0loss/FRt1/assignment_loss‚1.2443935175736744¬à€óE=€ ‚(:² ­ñôÒŽ®ä¾ disk.md0.out‚ 33.81640625 disk.dm-0.in‚0" disk.dm-0.out‚12481.9072265625 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚8124.1083984375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚5623.2744140625$ disk.nvme1n1.out‚5623.2744140625$ memory_percent‚96.37770652770996# proc.memory.rssMB‚ 2951.19140625 cpu‚25.856273754155275 network.sent‚ 2.992726e+06% disk./.usageGB‚1174.7346878051758' proc.memory.availableMB‚ 593.4765625 proc.cpu.threads‚17 network.recv‚418380) disk./.usagePercent‚76.51748495718455! disk.dm-1.out‚1192.3466796875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚8124.1083984375* proc.memory.percent‚18.012642860412598 disk.md0.in‚0ªtð|“ ‚(:ˆ ­ñôÒ¾¦À¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚121.872' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.624 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”§À8‚ ‚(:­ ¼ñôÒïÒؾ) disk./.usagePercent‚76.51748744192467 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚5624.00537109375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.012452125549316 network.sent‚ 3.055097e+06 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚8125.0380859375% disk.nvme1n1.out‚5624.00537109375 disk.md0.out‚ 34.203125" disk.dm-0.out‚12482.5166015625" disk.dm-1.out‚1192.55419921875 disk.rbd0.out‚0 network.recv‚422397% disk./.usageGB‚1174.7347259521484 disk.dm-0.in‚0$ memory_percent‚96.37908935546875" proc.memory.availableMB‚593.25# proc.memory.rssMB‚ 2951.16015625$ disk.nvme0n1.out‚8125.0380859375 disk.nvme1c1n1.in‚0 cpu‚25.85360564812881 proc.cpu.threads‚17çRd“ƒ ‚(:ˆ ¼ñôÒûÓ—¿ gpu.0.gpu‚71.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.113' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.371 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08û2„ - _wandbruntime‚1437  _runtime‚1437†§úLÉ… à _runtime‚1437.541272297 _step‚173 ' lr-AdamW/pg1‚0.00015731838442562186 ' lr-AdamW/pg2‚0.00015731838442562186 trainer/global_step‚2149 % _timestamp‚1.7844942809381752e+09åœÎ† È ' lr-AdamW/pg1‚0.00015731838442562186 ' lr-AdamW/pg2‚0.00015731838442562186 trainer/global_step‚2149 % _timestamp‚1.7844942809381752e+09 _runtime‚1437.541272297 _step‚173­ ObB2‡ - _wandbruntime‚1437  _runtime‚1437 X3W&ˆ ÚO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _step‚174 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1765082070944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21612445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126581090525184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015323 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168343349888512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166573405270528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168343244189184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16279272 trainer/global_step‚2149 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168343244189184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168343244189184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _step‚174 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1765082070944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21612445 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126581090525184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015323 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168343349888512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166573405270528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168343244189184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16279272®‰».<2Š - _wandbruntime‚1437  _runtime‚1437óêÙðê&‹ äM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1770730340864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21612122 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166648537897472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164684692959424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16286457 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4868425 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38645231 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21622279 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21622279 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17022952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4868244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 _runtime‚1437.943187092 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21612122 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1770641673728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164684675920064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124814431153664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1765865565656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16286272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770641673728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17022950 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38635072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159€Tªmï&Œ éM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164684692959424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16286457 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4868425 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38645231 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21622279 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21622279 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17022952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4868244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 _runtime‚1437.943187092 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21612122 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1770641673728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164684675920064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124814431153664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1765865565656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16286272 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770641673728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17022950 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38635072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1770730340864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21612122 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166648537897472¯rzD&2 - _wandbruntime‚1437  _runtime‚1437åb˜äŽ ü /loss/FRt2/assignment_loss‚2.800005766185554 2loss/SRqqt1/assignment_loss‚0.7616066239884853 0loss/FBt1/assignment_loss‚0.0940379148219866 &loss/total_loss‚13.537421361856945 0loss/FRt1/assignment_loss‚1.3224404143352135 trainer/global_step‚2149 _step‚176 /loss/SRqqt1/detection_loss‚0.73320963613391 .loss/FBt2/detection_loss‚1.963736036124816 epoch‚5 _runtime‚1437.943631053 2loss/SRqqt2/assignment_loss‚0.8575457756797021 1loss/FBt2/assignment_loss‚0.32554102253757894 /loss/FRt1/detection_loss‚1.5329961763681186 /loss/FRt2/detection_loss‚0.6123132988933877 0loss/SRqqt2/detection_loss‚1.327234556325288 /loss/FBt1/detection_loss‚1.2067541404629025 % _timestamp‚1.7844942813420942e+09ÀCœA  /loss/FRt2/detection_loss‚0.6123132988933877 0loss/SRqqt2/detection_loss‚1.327234556325288 /loss/FBt1/detection_loss‚1.2067541404629025 % _timestamp‚1.7844942813420942e+09 /loss/FRt2/assignment_loss‚2.800005766185554 2loss/SRqqt1/assignment_loss‚0.7616066239884853 0loss/FBt1/assignment_loss‚0.0940379148219866 &loss/total_loss‚13.537421361856945 0loss/FRt1/assignment_loss‚1.3224404143352135 trainer/global_step‚2149 _step‚176 /loss/SRqqt1/detection_loss‚0.73320963613391 .loss/FBt2/detection_loss‚1.963736036124816 epoch‚5 _runtime‚1437.943631053 2loss/SRqqt2/assignment_loss‚0.8575457756797021 1loss/FBt2/assignment_loss‚0.32554102253757894 /loss/FRt1/detection_loss‚1.5329961763681186°3ñ;`  ‚(:´ ËñôÒ¶„¿' proc.memory.availableMB‚ 592.9296875 cpu‚25.850804429776428 disk.rbd0.out‚0 network.recv‚425881) disk./.usagePercent‚76.51709584688213 disk.md0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚96.38104438781738 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚8126.73486328125 disk.nvme0n1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.071654e+06% disk./.usageGB‚1174.7287139892578 disk.nvme1n1.in‚0% disk.nvmeä­äÑ21n1.out‚5624.73583984375 disk.dm-1.in‚0% disk.nvme0n1.out‚8126.73486328125# proc.memory.rssMB‚ 2951.16015625* proc.memory.percent‚18.012452125549316 disk.md0.out‚34.5625# disk.dm-0.out‚12484.12744140625" disk.dm-1.out‚1192.60498046875' disk.nvme1c1n1.out‚5624.73583984375Ž†jd’‘ ‚(:‡ ËñôÒÉí²¿ gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚229.167' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚76.389 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰D¯L=’ ‚(:² ÚñôÒþ¢“¿$ disk.nvme0n1.out‚8127.7744140625 disk.nvme1c1n1.in‚0# disk.dm-0.out‚12484.83056640625* proc.memory.percent‚18.012452125549316 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚5625.4970703125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 34.98828125& proc.memory.availableMB‚ 592.890625% disk./.usageGB‚1174.7287826538086 disk.dm-0.in‚0$ memory_percent‚96.38128280639648# proc.memory.rssMB‚ 2951.16015625 proc.cpu.threads‚17 network.recv‚430120) disk./.usagePercent‚76.51710031941434& disk.nvme0c0n1.out‚8127.7744140625 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚5625.4970703125 cpu‚25.84823904685057 network.sent‚ 3.122999e+06 disk.dm-1.in‚0" disk.dm-1.out‚1192.78759765625ù솟“ ‚(:” Úñôҫÿ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚159.712' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚53.237333333333325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñPž2” - _wandbruntime‚1469  _runtime‚1469p}êÉ• à ' lr-AdamW/pg2‚0.00015713801942865256 trainer/global_step‚2199 % _timestamp‚1.7844943123998702e+09 _runtime‚1469.001675488 _step‚177 ' lr-AdamW/pg1‚0.00015713801942865256HÕtΖ È _step‚177 ' lr-AdamW/pg1‚0.00015713801942865256 ' lr-AdamW/pg2‚0.00015713801942865256 trainer/global_step‚2199 % _timestamp‚1.7844943123998702e+09 _runtime‚1469.001675488±v†ÉÔ2— - _wandbruntime‚1469  _runtime‚1469ï.ˆß'˜ ÙO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170330871546368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4973252 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1809294558208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170129249305384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚188 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127565085715968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1863483173376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.784494312400738e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1469.002499509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170330888585728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127565179825664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22095154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1809294558208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129428662999040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168324911664272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22105297 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1809205898240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16639216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1804233815326 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39491925 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11665776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17396773 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172140077444608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170330888585728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129428445736960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _step‚178 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172140077444608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080Üasä'™ ÞO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1809294558208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170129249305384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚188 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127565085715968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1863483173376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.784494312400738e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1469.002499509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170330888585728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127565179825664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22095154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1809294558208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129428662999040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168324911664272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22105297 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1809205898240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16639216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1804233815326 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39491925 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11665776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17396773 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172140077444608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170330888585728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129428445736960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _step‚178 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172140077444608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170330871546368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4973252 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584²ôr…ª2š - _wandbruntime‚1469  _runtime‚1469"<Z&› ãM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168399202189664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 % _timestamp‚1.7844943128024125e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172216118533120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170204323325488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39509374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129485389422080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127621223443968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170406038252032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17404400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4975378 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129485615065600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1805017309926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22115131 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚ 1469.40440202 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39519533 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16646206 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _step‚179 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170204219499590 trainer/global_step‚2199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168399219229024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1805104096464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170406021212672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4975561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11670832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11670828 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170406038252032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172216118533120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127621317553664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1864165978112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22115131 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172216012826624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16646393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17404400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1810080281088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end¡LÍ/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130;.úåî&œ èM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17404400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4975378 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129485615065600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1805017309926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22115131 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚ 1469.40440202 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39519533 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16646206 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _step‚179 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170204219499590 trainer/global_step‚2199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168399219229024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1805104096464 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170406021212672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4975561 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11670832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11670828 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170406038252032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172216118533120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127621317553664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1864165978112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22115131 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172216012826624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16646393 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17404400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1810080281088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168399202189664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 % _timestamp‚1.7844943128024125e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172216118533120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170204323325488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39509374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129485389422080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127621223443968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170406038252032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760³³ï°2 - _wandbruntime‚1469  _runtime‚1469A« ž ÿ 2loss/SRqqt1/assignment_loss‚0.6952601955102451 2loss/SRqqt2/assignment_loss‚0.8539830033100189 1loss/SRqqt1/detection_loss‚0.6897490126632062 0loss/SRqqt2/detection_loss‚1.297472477700829 epoch‚5 _runtime‚ 1469.40490843 _step‚180 0loss/FRt2/assignment_loss‚2.8429379251367126 1loss/FBt2/assignment_loss‚0.16147081950767936 /loss/FRt1/detection_loss‚1.4439791476317128 trainer/global_step‚2199 % _timestamp‚1.7844943128033535e+09 0loss/FRt1/assignment_loss‚1.2019018756970077 /loss/FBt2/detection_loss‚0.6881492663224661 &loss/total_loss‚13.294398295144973 1loss/FBt1/assignment_loss‚0.05090220934835874 /loss/FRt2/detection_loss‚0.5288380892428479 .loss/FBt1/detection_loss‚2.839754273073889Ìù;¤ Ÿ „ 2loss/SRqqt1/assignment_loss‚0.6952601955102451 2loss/SRqqt2/assignment_loss‚0.8539830033100189 1loss/SRqqt1/detection_loss‚0.6897490126632062 0loss/SRqqt2/detection_loss‚1.297472477700829 epoch‚5 _runtime‚ 1469.40490843 _step‚180 0loss/FRt2/assignment_loss‚2.8429379251367126 1loss/FBt2/assignment_loss‚0.16147081950767936 /loss/FRt1/detection_loss‚1.4439791476317128 trainer/global_step‚2199 % _timestamp‚1.7844943128033535e+09 0loss/FRt1/assignment_loss‚1.2019018756970077 /loss/FBt2/detection_loss‚0.6881492663224661 &loss/total_loss‚13.294398295144973 1loss/FBt1/assignment_loss‚0.05090220934835874 /loss/FRt2/detection_loss‚0.5288380892428479 .loss/FBt1/detection_loss‚2.839754273073889´®ºDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40376372 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172039454973232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16998744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5080351 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130371972115968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚2249 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40366227 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17778223 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17778223 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132276055312384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174088371900928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11918572 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22588006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1848555838464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175936910700032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17778221 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11918576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1848555838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843472346134 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175937016399360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5080172 _step‚182 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175936910700032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1848644498432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174088354861568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22598149 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174088354861568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1903861740032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172039437933872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843559132560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132275833856000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130372066225664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚1500.472299712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22588006 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175937016399360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 % _timestamp‚1.7844943438693995e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22598149 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0LþíC« ßO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174088354861568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22598149 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174088354861568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1903861740032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172039437933872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843559132560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132275833856000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130372066225664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚1500.472299712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22588006 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175937016399360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 % _timestamp‚1.7844943438693995e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22598149 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40376372 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172039454973232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16998744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5080351 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130371972115968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚2249 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40366227 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17778223 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17778223 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132276055312384 D>DeviceStatsMonitor.on_train ‹Æ_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174088371900928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11918572 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22588006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1848555838464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175936910700032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17778221 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11918576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1848555838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843472346134 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175937016399360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5080172 _step‚182 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175936910700032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1848644498432¶=ëz2¬ - _wandbruntime‚1500  _runtime‚1500†/½_ê&­ äM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174163521567232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176012846082048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17006017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22597826 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174163504527872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173957984299998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5082211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1904798551040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173958088125896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176012951788544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22597826 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5082385 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132333002504704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1849430221312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17785852 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844943442711296e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40383676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132332774764032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚2249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 _step‚183 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172113728459264 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40393835 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17785850 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130428203953664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17005839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172113745498624«¹¹ï&® éM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17785852 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17785850 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22607983 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1849341554176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1849341554176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174163504527872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174163521567232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22607983 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173957984299998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5082211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1904798551040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173958088125896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176012951788544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22597826 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5082385 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132333002504704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1849430221312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17785852 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844943442711296e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40383676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132332774764032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚2249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 _step‚183 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172113728459264 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40393835 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17785850 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130428203953664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17005839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172113745498624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174163521567232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176012846082048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17006017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22597826 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174163504527872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18159673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177845855216128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚2299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚167 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133178952625664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177845838176768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179733743955456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12171376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1944362590208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1882710876942 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚1531.619278406 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1887994438656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18159673 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7844943750137663e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18159671 M=DeviceStatsMonitor.on_train_batch_start/active_byte|÷Ds.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18159671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135123440465408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175753981242832 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23091001 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177636778906200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179733743955456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1882797663368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179733849654784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1887905778688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5186430 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177845838176768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18159673 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177845855216128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚2299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚167 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133178952625664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177845838176768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179733743955456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12171376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1944362590208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1882710876942 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _runtime‚1531.619278406 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1887994438656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18159673 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7844943750137663e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18159671 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18159671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135123440465408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175753981242832 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23091001 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177636778906200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179733743955456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1882797663368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179733849654784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1887905778688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5186430 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177845838176768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135180383963136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175828271768224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1888780161536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41268137 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18167302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5188803 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17365059 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18167300 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23100835 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23090678 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41257978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12176432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18167300 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175828254728864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41268137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133234996243968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17365235 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18167302 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177711749100406 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177921004882432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1945293609472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7844943754155087e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1883494371542 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚172 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177711852926304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12176428 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179809785043968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177921004882432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177920987843072 sbpܾ éM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7844943754155087e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1883494371542 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚172 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177711852926304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12176428 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179809785043968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177921004882432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177920987843072 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135180383963136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175828271768224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1888780161536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41268137 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18167302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5188803 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17365059 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18167300 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23100835 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23090678 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41257978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12176432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18167300 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175828254728864 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41268137 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.ÁÄî^ on_train_batch_end/inactive_split.large_pool.current‚4 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133234996243968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 _step‚187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17365235 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18167302 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177711749100406 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177921004882432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1945293609472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BÜ2¿ - _wandbruntime‚1532  _runtime‚1532/B„±À  /loss/FRt2/detection_loss‚0.6423150497283341 /loss/FBt2/detection_loss‚0.8839406858498832 &loss/total_loss‚11.060701418800143 % _timestamp‚1.7844943754164734e+09 _step‚188 0loss/FRt2/assignment_loss‚2.5299239950501815 1loss/FBt2/assignment_loss‚0.13510568335974607 /loss/FBt1/detection_loss‚0.9344865050317203 2loss/FBt1/assignment_loss‚0.023292219864523112 /loss/FRt1/detection_loss‚1.4251765336842201 1loss/SRqqt1/detection_loss‚0.7460676959791023 1loss/SRqqt2/assignment_loss‚0.780369877628982 0loss/SRqqt2/detection_loss‚1.152542639792664 trainer/global_step‚2299 _runtime‚1532.018009506 epoch‚5 0loss/FRt1/assignment_loss‚1.0848615837886053 2loss/SRqqt1/assignment_loss‚0.7226189490421816÷ÕnŠ Á † 2loss/FBt1/assignment_loss‚0.023292219864523112 /loss/FRt1/detection_loss‚1.4251765336842201 1loss/SRqqt1/detection_loss‚0.7460676959791023 1loss/SRqqt2/assignment_loss‚0.780369877628982 0loss/SRqqt2/detection_loss‚1.152542639792664 trainer/global_step‚2299 _runtime‚1532.018009506 epoch‚5 0loss/FRt1/assignment_loss‚1.0848615837886053 2loss/SRqqt1/assignment_loss‚0.7226189490421816 /loss/FRt2/detection_loss‚0.6423150497283341 /loss/FBt2/detection_loss‚0.8839406858498832 &loss/total_loss‚11.060701418800143 % _timestamp‚1.7844943754164734e+09 _step‚188 0loss/FRt2/assignment_loss‚2.5299239950501815 1loss/FBt2/assignment_loss‚0.13510568335974607 /loss/FBt1/detection_loss‚0.9344865050317203¼â‡ëF@ ‚(:µ ´òôÒÁçý¾ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 network.sent‚ 3.265463e+06) disk./.usagePercent‚76.51912886124639 disk.nvme0n1.in‚0% disk.nvme0n1.out‚8141.10400390625# proc.memory.rssMB‚ 2952.05078125 proc.cpu.threads‚17 network.recv‚451680% disk./.usageGB‚1174.7599258422852 disk.md0.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 592.49609375* proc.memory.percent‚18.017888069152832 cpu‚25.833432184934516 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚5629.2412109375 disk.md0.out‚ 36.9140625# disk.dm-0.out‚12497.10595703125! disk.dm-1.out‚1193.4150390625' disk.nvme0c0n1.out‚8141.10400390625$ memory_percent‚96.38369083404541& disk.nvme1c1n1.out‚5629.2412109375 disk.dm-0.in‚0åjkoœà ‚(:‘ ´òôÒÛÍÝ¿ gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.003' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.33433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šª 37Ä ‚(:¬ ÃòôÒæï;" disk.dm-0.out‚12514.8876953125 network.recv‚456215 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0" disk.nvme0n1.out‚ 8158.16015625' disk.nvme1c1n1.out‚5630.95263671875$ memory_percent‚96.38371467590332 proc.cpu.threads‚17" disk.dm-1.out‚1193.54736328125$ disk.nvme0c0n1.out‚ 8158.16015625 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 592.4921875# proc.memory.rssMB‚ 2952.05078125* proc.memory.percent‚18.017888069152832 cpu‚25.83137927911062) disk./.usagePercent‚76.34211945390103% disk./.usageGB‚1172.0423889160156 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚37.3125 disk.dm-0.in‚0 network.sent‚ 3.327892e+06 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚5630.95263671875 disk.md0.in‚0+ÀtWžÅ ‚(:“ ÃòôÒ™¿‘¿ gpu.0.gpu‚31.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚137.454' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚45.818000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øIhm2Æ - _wandbruntime‚1563  _runtime‚1563Ù[¿ÉÇ Ã ' lr-AdamW/pg2‚0.00015656733253022832 trainer/global_step‚2349 % _timestamp‚1.7844944064411209e+09 _runtime‚1563.042952005 _step‚189 ' lr-AdamW/pg1‚0.00015656733253022832¡U™DÎÈ È % _timestamp‚1.7844944064411209e+09 _runtime‚1563.042952005 _step‚189 ' lr-AdamW/pg1‚0.00015656733253022832 ' lr-AdamW/pg2‚0.00015656733253022832 trainer/global_step‚2349½ååÖ2É - _wandbruntime‚1563  _runtime‚1563ìÒ à'Ê ÚO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚166 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5293814 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 trainer/global_step‚2349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181603338531328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17717986 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116860928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179468490473072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42124976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _runtime‚1563.047259451 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18541123 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135985839025664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18541121 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1984978522112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18541123 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1927344378880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927255718912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181603321491968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183530577210880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚162 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1922036194176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12424176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23573710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183530577210880 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181390543706608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5293976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137970606577152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23573710 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1927255718912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181390439880822 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23583853 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137970817547776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17718152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18541121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985744915968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1927344378880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12424172 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42114831 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181603321491968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179468507512432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1921949407750 % _timestamp‚1.7844944064419816e+09 _step‚190 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183530682910208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17718152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18541121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985744915968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1927344378880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12424172 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42114831 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181603321491968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179468507512432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1921949407750 % _timestamp‚1.7844944064419816e+09 _step‚190 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183530682910208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18541123 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1927344378880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927255718912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181603321491968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183530577210880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚162 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1922036194176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12424176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23573710 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183530577210880 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181390543706608 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5293976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137970606577152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23573710 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760¾|N 2Ì - _wandbruntime‚1563  _runtime‚1563VB'ö•Í äM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136041882643968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181465513900814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12429232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42142439 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183606618299392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18548750 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18548750 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12429228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5296043 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181678471158272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138027765919744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _step‚191 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17725114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42132280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928041434624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5295886 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183606512592896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 trainer/global_step‚2349 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928041434624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7844944068445668e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18548752 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42142439 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1985670215168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138027552859136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179542780998464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1922819688888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181678471158272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1928130101760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1922732902350 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181678488197632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181678488197632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23593687 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚1618gv+ï&Î éM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181678488197632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181678488197632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23593687 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚161 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136041882643968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181465513900814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12429232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42142439 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183606618299392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18548750 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18548750 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12429228 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5296043 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181678471158272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138027765919744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _step‚191 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17725114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42132280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928041434624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5295886 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183606512592896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 trainer/global_step‚2349 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928041434624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7844944068445668e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18548752 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42142439 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1985670215168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138027552859136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179542780998464 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1922819688888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181678471158272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1928130101760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1922732902350¿õ ×C2Ï - _wandbruntime‚1563  _runtime‚1563ýUãôÐ € trainer/global_step‚2349 % _timestamp‚1.7844944068455002e+09 0loss/FRt2/assignment_loss‚2.6518868284266177 2loss/SRqqt2/assignment_loss‚0.7741706808239287 1loss/SRqqt2/detection_loss‚1.1806105101765816 epoch‚5 _step‚192 /loss/FRt1/detection_loss‚1.5208146122336283 &loss/total_loss‚11.793626967737085 2loss/FBt1/assignment_loss‚0.059305302049299374 1loss/FBt2/assignment_loss‚0.13105757214733896 /loss/FRt2/detection_loss‚0.6472216521864016 0loss/SRqqt1/detection_loss‚0.770077692084891 _runtime‚1563.447054397 /loss/FRt1/assignment_loss‚1.121982526874352 2loss/SRqqt1/assignment_loss‚0.7062052201343242 .loss/FBt1/detection_loss‚1.004745077479165 /loss/FBt2/detection_loss‚1.2255492931205565¸æº Ñ … /loss/FRt1/detection_loss‚1.5208146122336283 &loss/total_loss‚11.793626967737085 2loss/FBt1/assignment_loss‚0.059305302049299374 1loss/FBt2/assignment_loss‚0.13105757214733896 /loss/FRt2/detection_loss‚0.6472216521864016 0loss/SRqqt1/detection_loss‚0.770077692084891 _runtime‚1563.447054397 /loss/FRt1/assignment_loss‚1.121982526874352 2loss/SRqqt1/assignment_loss‚0.7062052201343242 .loss/FBt1/detection_loss‚1.004745077479165 /loss/FBt2/detection_loss‚1.2255492931205565 trainer/global_step‚2349 % _timestamp‚1.7844944068455002e+09 0loss/FRt2/assignment_loss‚2.6518868284266177 2loss/SRqqt2/assignment_loss‚0.7741706808239287 1loss/SRqqt2/detection_loss‚1.1806105101765816 epoch‚5 _step‚192ÀþEy“<Ò ‚(:± ÒòôÒßÒð¾' disk.nvme1c1n1.out‚5631.68017578125$ memory_percent‚96.77705764770508 disk.nvme0c0n1.in‚0 disk.md0.out‚ 37.7890625 disk.dm-0.in‚0* proc.memory.percent‚18.017888069152832 proc.cpu.threads‚17 disk.nvme1n1.in‚0 network.sent‚ 3.332729e+06% disk./.usageGB‚1172.0424728393555 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0" disk.dm-0.out‚12520.0947265625 disk.dm-1.in‚0" disk.dm-1.out‚1193.66455078125( disk./.usagePercent‚76.3421249203293 disk.rbd0.in‚0$ disk.nvme0n1.out‚8163.7744140625% disk.nvme1n1.out‚5631.68017578125 disk.md0.in‚0& proc.memory.availableMB‚ 528.046875# proc.memory.rssMB‚ 2952.05078125 cpu‚25.82902822950905 network.recv‚459555 disk.rbd0.out‚0& disk.nvme0c0n1.out‚8163.7744140625[fÖÓ ‚(:’ ÒòôÒúÜο gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚227.57' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.85666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™k¸«BÔ ‚(:· áòôÒ™ú¾# disk.dm-0.out‚12544.80322265625" disk.dm-1.out‚1193.75146484375 disk.rbd0.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚8188.37841796875 network.sent‚ 3.395909e+06% disk./.usageGB‚1172.5432662963867' disk.nvme1c1n1.out‚5632.52978515625 disk.nvme1n1.in‚0 disk.md0.out‚ 38.08984375' disk.nvme0c0n1.out‚8188.37841796875$ memory_percent‚96.77858352661133# proc.memory.rssMB‚ 2952.05078125) disk./.usagePercent‚76.37474458859383 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0& proc.memory.availableMB‚ 527.796875* proc.memory.percent‚18.017888069152832 cpu‚25.826572062401755 proc.cpu.threads‚17 network.recv‚463895 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚5632.52978515625ëf’Õ ‚(:‡ áòôÒ‘Ûª¿ gpu.0.gpu‚15.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.026' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T5zA2Ö - _wandbruntime‚1594  _runtime‚1594!/º¥È× Â _runtime‚ 1594.51253239 _step‚193 ' lr-AdamW/pg1‚0.00015636728032183871 ' lr-AdamW/pg2‚0.00015636728032183871 trainer/global_step‚2399 % _timestamp‚1.7844944379106903e+096пÍØ Ç trainer/global_step‚2399 % _timestamp‚1.7844944379106903e+09 _runtime‚ 1594.51253239 _step‚193 ' lr-AdamW/pg1‚0.00015636728032183871 ' lr-AdamW/pg2‚0.00015636728032183871Á±™÷ú2Ù - _wandbruntime‚1594  _runtime‚1594‘Ó¸à'Ú ÚO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140818220678144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1966605659136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18077268 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24076705 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187327410466304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1966694319104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2025365808640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183183016742672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚185360804807168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12676972 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183183033782032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24066562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12676976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42999278 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18922573 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24066562 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1961274724984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1961187938558 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138792725425664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682¦Q‘ ŠÛ ßO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42999278 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18922573 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24066562 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1961274724984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1961187938558 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138792725425664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140818220678144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1966605659136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18077268 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 E;DeviceStatsMonitor.on_trai¿MÕ÷[n_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372523008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24076705 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187327410466304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1966694319104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2025365808640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183183016742672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚185360804807168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12676972 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183183033782032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24066562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12676976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Â-\%2Ü - _wandbruntime‚1594  _runtime‚1594!gê&Ý äM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24076382 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1967480041984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚162 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187403451554816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185435954473472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185435954473472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43006582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140874943144448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138848863153664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _runtime‚1594.915852955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18930202 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 % _timestamp‚1.7844944383138719e+09 _step‚195 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24086539 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43016741 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2026174100480 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1967391374848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1967391374848 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18930200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1961971433158 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185219278701222 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183257307268064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185435971512832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18930200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187403451554816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2026307731456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5402557 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24076382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140875170885120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185219382527120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43006582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1967480041984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185435971512832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12682028 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12682032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183257324307424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5402395 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1962058219696 trainer/global_step‚2399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18084423 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18084589 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43016741 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18930202 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187403345848320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24086539 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303óÜñ°ï&Þ éM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1967391374848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1967391374848 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18930200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1961971433158 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372523008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133630976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185219278701222 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183257307268064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185435971512832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18930200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187403451554816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2026307731456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5402557 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24076382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140875170885120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185219382527120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43006582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1967480041984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185435971512832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12682028 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12682032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183257324307424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5402395 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1962058219696 trainer/global_step‚2399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18084423 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18084589 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43016741 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18930202 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187403345848320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24086539 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24076382 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1967480041984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚162 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187403451554816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185435954473472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185435954473472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43006582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚166 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140874943144448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138848863153664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _runtime‚1594.915852955 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18930202 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 % _timestamp‚1.7844944383138719e+09 _step‚195 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24086539 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43016741 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2026174100480ÃYwÅo2ß - _wandbruntime‚1594  _runtime‚1594Cºžà  /loss/FRt2/detection_loss‚0.6424722631068479 /loss/FBt2/detection_loss‚1.3046329142300237 % _timestamp‚1.7844944383148377e+09 _runtime‚1594.917402687 _step‚196 0loss/FRt1/assignment_loss‚1.2041608546343114 trainer/global_step‚2399 2loss/SRqqt1/assignment_loss‚0.7280992311336312 1loss/FBt2/assignment_loss‚0.18863966555028072 /loss/FRt1/detection_loss‚1.3928617655508586 1loss/SRqqt1/detection_loss‚0.7191299985612875 1loss/SRqqt2/detection_loss‚1.2356408949463482 /loss/FBt1/detection_loss‚1.0065697340213624 %loss/total_loss‚12.10688782143253 epoch‚5 2loss/SRqqt2/assignment_loss‚0.8133160392835475 2loss/FBt1/assignment_loss‚0.059648762892704696 /loss/FRt2/assignment_loss‚2.811715697521325Œ +ã á † /loss/FRt2/assignment_loss‚2.811715697521325 /loss/FRt2/detection_loss‚0.6424722631068479 /loss/FBt2/detection_loss‚1.3046329142300237 % _timestamp‚1.7844944383148377e+09 _runtime‚1594.917402687 _step‚196 0loss/FRt1/assignment_loss‚1.2041608546343114 trainer/global_step‚2399 2loss/SRqqt1/assignment_loss‚0.7280992311336312 1loss/FBt2/assignment_loss‚0.18863966555028072 /loss/FRt1/detection_loss‚1.3928617655508586 1loss/SRqqt1/detection_loss‚0.7191299985612875 1loss/SRqqt2/detection_loss‚1.2356408949463482 /loss/FBt1/detection_loss‚1.0065697340213624 %loss/total_loss‚12.10688782143253 epoch‚5 2loss/SRqqt2/assignment_loss‚0.8133160392835475 2loss/FBt1/assignment_loss‚0.059648762892704696ÄW†Kë3â ‚(:¨ ðòôÒþ»á¾ disk.nvme0n1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0$ memory_percent‚96.78535461425781$ proc.memory.availableMB‚526.6875 disk.dm-0.in‚0! disk.dm-0.out‚13591.916015625 network.sent‚ 3.398831e+06( disk./.usagePercent‚76.5219845730636$ disk./.usageGB‚1174.803768157959 disk.md0.out‚ 38.37109375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 network.recv‚467435' disk.nvme1c1n1.out‚5633.02294921875" disk.dm-1.out‚1193.85302734375! disk.nvme0n1.out‚ 9235.7265625# proc.memory.rssMB‚ 2953.37109375* proc.memory.percent‚18.025946617126465 cpu‚25.84393684725219 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚5633.02294921875 disk.md0.in‚0# disk.nvme0c0n1.out‚ 9235.7265625ßaÒûã ‚(:’ ðòôÒ«·Â¿ gpu.0.gpu‚49.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚189.091' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.03033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\^|2ä - _wandbruntime‚1608  _runtime‚1608:*3å ë{ 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6802825927734375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04054354876279831 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5872677564620972 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8269230723381042 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9769303202629089 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2343265265226364 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7477929592132568 )jet/accuracy_2_of_4‚0.735608696937561 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2857474386692047 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021286824718117714 ,jet/accuracy_4_of_4‚0.011098379269242287 _runtime‚1608.136024567 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7000945210456848 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46522635221481323 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08561311662197113 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.594258189201355 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7828230857849121 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_3‚0.8924879431724548 /particle/accuracy_4_of_4‚0.94255²ÑMmë660235977173 epoch‚5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9960165619850159 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18388745188713074 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7109814882278442 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583432674408 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6139774322509766 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6155155897140503 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09633591026067734 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17783218622207642 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4307055175304413 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.736176609992981 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0536147765815258 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9948703050613403 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6344016790390015 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15079008042812347 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6082994937896729 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8147631287574768 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5957282185554504 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.683600664138794 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚11.330605078128652 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8842592239379883 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7477929592132568 *particle/sensitivity‚0.588455319404602 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9838307499885559 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6325485110282898 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02795887552201748 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2634825110435486 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5617756247520447 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21226927638053894 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6000624895095825 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6717590093612671 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24903392791748047 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892674684524536 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7407585382461548 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9744272828102112 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08965223282575607 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3564814627170563 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9879079461097717 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13676020503044128 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08166666328907013 particle/accuracy_2_of_2‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.29573991894721985 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7316575646400452 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5687934160232544 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48842594027519226 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21198590099811554 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.629693329334259 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19985073804855347 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9331996440887451 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6103808283805847 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.736321747303009 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12094131857156754 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7314499616622925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5827858448028564 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9580844044685364 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5628997087478638 _step‚197 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848607182502747 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8785157203674316 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49916666746139526 +jet/accuracy_2_of_2‚0.12500610947608948 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09059696644544601 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8378041386604309 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24930521845817566 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7763605713844299 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9881404638290405 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47059813141822815 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9670068621635437 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6202396154403687 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5700768232345581 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21119429171085358 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20068450272083282 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12738549709320068 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961200952529907 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6123255491256714 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16231699287891388 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5372904539108276 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5399644374847412 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5621473789215088 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791269302368 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13446888327598572 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961200952529907 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_2_of_3‚0.46992728114128113 +jet/accuracy_3_of_4‚0.08443120121955872 7validation_average_jet_accuracy‚0.33885109424591064 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19032630324363708 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5953418016433716 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.668465256690979 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9965591430664062 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9940115809440613 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09633591026067734 *jet/accuracy_1_of_4‚0.9472231864929199 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6423847675323486 (particle/accuracy‚0.8386772871017456 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1996964067220688 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740740299224854 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33746838569641113 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2876551151275635 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6212060451507568 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08732551336288452 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7686696648597717 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5327997803688049 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8715628385543823 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6193907260894775 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3735705018043518 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5687934160232544 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5287001132965088 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2173117697238922 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5345091819763184 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9870679974555969 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9665998816490173 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 trainer/global_step‚2399 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954648613929749 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8820232152938843 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03344206139445305 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2076970934867859 % _timestamp‚1.7844944515337813e+09 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950568079948425 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1891489326953888 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06294940412044525 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19375155866146088 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12975355982780457 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15294615924358368 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923384785652161 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20068450272083282 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33642610907554626 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5941189527511597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9844850897789001 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17708955705165863 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.2982247769832611 /particle/accuracy_3_of_4‚0.9874716401100159 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042598266154527664 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +particle/specificity‚0.9071048498153687 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5613871812820435 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9933862090110779 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1896100789308548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7771731019020081 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47828856110572815 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394Öj…ö=æ ð{ 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9960165619850159 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18388745188713074 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7109814882278442 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583432674408 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6139774322509766 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6155155897140503 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09633591026067734 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17783218622207642 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4307055175304413 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.736176609992981 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0536147765815258 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9948703050613403 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6344016790390015 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15079008042812347 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6082994937896729 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8147631287574768 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5957282185554504 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.683600664138794 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚11.330605078128652 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8842592239379883 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7477929592132568 *particle/sensitivity‚0.588455319404602 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9838307499885559 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6325485110282898 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02795887552201748 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2634825110435486 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5617756247520447 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21226927638053894 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6000624895095825 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6717590093612671 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24903392791748047 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892674684524536 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7407585382461548 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9744272828102112 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08965223282575607 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3564814627170563 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9879079461097717 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13676020503044128 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08166666328907013 particle/accuracy_2_of_2‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.29573991894721985 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7316575646400452 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5687934160232544 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48842594027519226 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21198590099811554 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.629693329334259 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19985073804855347 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9331996440887451 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6103808283805847 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.736321747303009 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12094131857156754 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7314499616622925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5827858448028564 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9580844044685364 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5628997087478638 _step‚197 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848607182502747 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8785157203674316 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49916666746139526 +jet/accuracy_2_of_2‚0.12500610947608948 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09059696644544601 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8378041386604309 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24930521845817566 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7763605713844299 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9881404638290405 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47059813141822815 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9670068621635437 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6202396154403687 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5700768232345581 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21119429171085358 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20068450272083282 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12738549709320068 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961200952529907 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6123255491256714 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16231699287891388 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5372904539108276 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5399644374847412 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5621473789215088 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791269302368 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13446888327598572 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961200952529907 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_2_of_3‚0.46992728114128113 +jet/accuracy_3_of_4‚0.08443120121955872 7validation_average_jet_accuracy‚0.33885109424591064 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19032630324363708 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5953418016433716 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.668465256690979 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9965591430664062 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9940115809440613 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09633591026067734 *jet/accuracy_1_of_4‚0.9472231864929199 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6423847675323486 (particle/accuracy‚0.8386772871017456 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1996964067220688 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740740299224854 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33746838569641113 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2876551151275635 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6212060451507568 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08732551336288452 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7686696648597717 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5327997803688049 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8715628385543823 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6193907260894775 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3735705018043518 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5687934160232544 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5287001132965088 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2173117697238922 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5345091819763184 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9870679974555969 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9665998816490173 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 trainer/global_step‚2399 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954648613929749 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8820232152938843 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03344206139445305 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2076970934867859 % _timestamp‚1.7844944515337813e+09 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950568079948425 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1891489326953888 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06294940412044525 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19375155866146088 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12975355982780457 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15294615924358368 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923384785652161 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20068450272083282 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33642610907554626 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5941189527511597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9844850897789001 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17708955705165863 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.2982247769832611 /particle/accuracy_3_of_4‚0.9874716401100159 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042598266154527664 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +particle/specificity‚0.9071048498153687 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5613871812820435 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9933862090110779 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1896100789308548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7771731019020081 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47828856110572815 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6802825927734375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04054354876279831 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5872677564620972 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8269230723381042 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9769303202629089 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2343265265226364 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7477929592132568 )jet/accuracy_2_of_4‚0.735608696937561 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2857474386692047 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021286824718117714 ,jet/accuracy_4_of_4‚0.011098379269242287 _runtime‚1608.136024567 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7000945210456848 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46522635221481323 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08561311662197113 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.594258189201355 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7828230857849121 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_3‚0.8924879431724548 /particle/accuracy_4_of_4‚0.9425560235977173 epoch‚5Åýuwç ‚(:’ ÿòôÒªÜÀ gpu.0.gpu‚68.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.33' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.11000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{´†‡;è ‚(:° ÿòôÒžàƒÃ% disk./.usageGB‚1174.8241920471191 disk.nvme1n1.in‚0 disk.dm-1.out‚1293.916015625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚10894.8564453125) disk./.usagePercent‚76.52331490292259$ disk.nvme1n1.out‚5852.5732421875 disk.dm-0.in‚0 disk.dm-0.out‚15369.66796875( proc.memory.availableMB‚ 526.65234375# proc.memory.rssMB‚ 2953.37109375 cpu‚25.847320300138296 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 38.77734375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚10894.8564453125 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚5852.5732421875 disk.dm-1.in‚0" memory_percent‚96.785569190979* proc.memory.percent‚18.025946617126465 network.sent‚ 3.462131e+06 network.recv‚471551“¾Ò5é ‚(:ª ŽóôÒå؈¿ network.recv‚475611) disk./.usagePercent‚76.52337503363344 disk.nvme0n1.in‚0% disk.nvme1c1n1.out‚5881.056640625 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 526.64453125 cpu‚25.844946817231744# disk.nvme0n1.out‚10895.82421875 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚5881.056640625 disk.md0.in‚0 disk.md0.out‚ 39.10546875 disk.dm-1.out‚1321.857421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2953.37109375 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚10895.82421875 disk.dm-0.out‚ 15370.453125 disk.rbd0.out‚0* proc.memory.percent‚18.025946617126465 proc.cpu.threads‚17 network.sent‚ 3.518102e+06% disk./.usageGB‚1174.8251152038574 disk.dm-0.in‚0 disk.dm-1.in‚0$ memory_percent‚96.78561687469482ÅAþ§ê ‚(:’ ŽóôÒØãÈ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.472' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.157333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜*€…2ë - _wandbruntime‚1639  _runtime‚1639ÅîöÆì À _runtime‚1639.784313583 _step‚198 & lr-AdamW/pg1‚0.0001561623376447143 & lr-AdamW/pg2‚0.0001561623376447143 trainer/global_step‚2449 $ _timestamp‚1.784494483181012e+09Ü°›>í Å $ _timestamp‚1.784494483181012e+09 _runtime‚163cÛÜ9.784313583 _step‚198 & lr-AdamW/pg1‚0.0001561623376447143 & lr-AdamW/pg2‚0.0001561623376447143 trainer/global_step‚2449Ƴ+Z2î - _wandbruntime‚1639  _runtime‚1639l‹öWà'ï ÚO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192136160394752 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187871027087824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24652064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2015841727932 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24641921 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142402977030656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190114775418368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192136160394752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19425821 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19425821 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19425823 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192136266094080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7844944831815958e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5555774 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190114775418368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2021473636352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144484411711488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118958080 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2015928514358 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192136160394752 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187871027087824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24652064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2015841727932 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24641921 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142402977030656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190114775418368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192136160394752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19425821 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19425821 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19425823 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192136266094080ÇS†ßÞ2ñ - _wandbruntime‚1640  _runtime‚1640ËÞ?îê&ò äM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190189925084672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚192212095776768 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18600779 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189961925796388 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190189925084672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5557882 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _runtime‚1640.184891516 _step‚200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2016712009070 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44095350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142459114758656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192212201483264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190189942124032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44085191 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13043087 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189962029622286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2016625222532 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022259359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 trainer/global_step‚2449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2022170692096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5557696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187945300573856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187945300573856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18600779 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚189961925796388 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190189925084672 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5557882 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _runtime‚1640.184891516 _step‚200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2016712009070 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44095350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142459114758656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192212201483264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190189942124032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44085191 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13043087 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚189962029622286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2016625222532 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022259359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 trainer/global_step‚2449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682ÈÏt2ô - _wandbruntime‚1640  _runtime‚1640ä=ŸEõ ÿ /loss/FRt2/assignment_loss‚2.610347798218585 2loss/SRqqt1/assignment_loss‚0.6664637817125167 0loss/FBt1/assignment_loss‚0.0931658470618578 0loss/FBt2/assignment_loss‚0.2030078681270362 1loss/SRqqt1/detection_loss‚0.8451364595790314 _step‚201 0loss/FRt1/assignment_loss‚1.1678846132875693 /loss/FRt1/detection_loss‚1.4393960654019435 /loss/FRt2/detection_loss‚0.5410764514121518 1loss/SRqqt2/detection_loss‚1.1301886029434267 &loss/total_loss‚11.279897893130277 /loss/FBt1/detection_loss‚0.8474356856909084 epoch‚6 2loss/SRqqt2/assignment_loss‚0.8062543621253051 /loss/FBt2/detection_loss‚0.9295403575699451 trainer/global_step‚2449 % _timestamp‚1.7844944835838397e+09 _runtime‚1640.185409497ñe¨O ö „ 0loss/FBt2/assignment_loss‚0.2030078681270362 1loss/SRqqt1/detection_loss‚0.8451364595790314 _step‚201 0loss/FRt1/assignment_loss‚1.1678846132875693 /loss/FRt1/detection_loss‚1.4393960654019435 /loss/FRt2/detection_loss‚0.5410764514121518 1loss/SRqqt2/detection_loss‚1.1301886029434267 &loss/total_loss‚11.279897893130277 /loss/FBt1/detection_loss‚0.8474356856909084 epoch‚6 2loss/SRqqt2/assignment_loss‚0.8062543621253051 /loss/FBt2/detection_loss‚0.9295403575699451 trainer/global_step‚2449 % _timestamp‚1.7844944835838397e+09 _runtime‚1640.185409497 /loss/FRt2/assignment_loss‚2.610347798218585 2loss/SRqqt1/assignment_loss‚0.6664637817125167 0loss/FBt1/assignment_loss‚0.0931658470618578É(@0¥9÷ ‚(:® óôÒ­€…¿ disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5882.189453125) disk./.usagePercent‚76.52336434925093$ disk./.usageGB‚1174.824951171875 disk.dm-1.in‚0& disk.nvme0n1.out‚10912.98193359375 disk.nvme1n1.in‚0# disk.nvme1n1.out‚5882.189453125 disk.md0.in‚0 disk.md0.out‚ 39.48046875# disk.dm-0.out‚15387.76904296875( disk.nvme0c0n1.out‚10912.98193359375# proc.memory.rssMB‚ 2953.37109375 cpu‚25.842739544913663 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.rbd0.in‚0$ memory_percent‚96.78606986999512' proc.memory.availableMB‚ 526.5703125* proc.memory.percent‚18.025946617126465 network.sent‚ 3.53571e+06 network.recv‚479102 disk.dm-1.out‚ 1322.03125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0Ðò¯žø ‚(:“ óôÒ‚Š¸¿ gpu.0.gpu‚98.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚115.316' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.43866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆa±:8ù ‚(:­ ¬óôÒ޺̾ disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10914.08154296875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 40.03515625$ memory_percent‚96.78606986999512 network.sent‚ 3.585289e+06 disk.dm-0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 526.5703125# proc.memory.rssMB‚ 2953.37109375 network.recv‚483380% disk./.usageGB‚1174.8249397277832# disk.dm-0.out‚15388.40576171875 disk.dm-1.out‚ 1322.109375 disk.nvme0n1.in‚0$ disk.nvme1c1n1.out‚ 5882.96484375* proc.memory.percent‚18.025946617126465 cpu‚25.84043616515197( disk./.usagePercent‚76.5233636038289 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚10914.08154296875" disk.nvme1n1.out‚ 5882.96484375 proc.cpu.threads‚17þ®±\ú ‚(:’ ¬óôÒ×ꎿ gpu.0.gpu‚54.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚223.166' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.38866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úiªø2û - _wandbruntime‚1671  _runtime‚1671ÝÐ# Èü  _runtime‚ 1671.24971054 _step‚202 ' lr-AdamW/pg1‚0.00015595251766194017 ' lr-AdamW/pg2‚0.00015595251766194017 trainer/global_step‚2499 % _timestamp‚1.7844945146478555e+09ì.Þ/Íý Ç ' lr-AdamW/pg1‚0.00015595251766194017 ' lr-AdamW/pg2‚0.00015595251766194017 trainer/global_step‚2499 % _timestamp‚1.7844945146478555e+09 _runtime‚ 1671.24971054 _step‚202ÊFð'2þ - _wandbruntime‚1671  _runtime‚1671ŽÞAà'ÿ ÚO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193872275772928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25144916 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193872258733568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2499 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193640616576804 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2055080258740 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚193872258733568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚203 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2060734916608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195932993650176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2060734916608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19807273 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807271 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2055167045166 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195932993650176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147332152741888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2122178741760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25134773 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2060823576576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191585536318064 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚195933099349504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2060823576576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147331948062720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193872275772928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 110569472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18955039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191585553357424KGÇ'€ ßO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚193872258733568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚203 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2060734916608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195932993650176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2060734916608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19807273 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807271 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2055167045166 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195932993650176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147332152741888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2122178741760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25134773 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2060823576576 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191585536318064 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚195933099349504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2060823576576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147331948062720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193872275772928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 110569472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18955039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191585553357424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193872275772928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25144916 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193872258733568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2499 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.fÑ/™5Íreed‚193640616576804 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2055080258740 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760ËŽ©àÜ2 - _wandbruntime‚1671  _runtime‚1671Íð*›ê&‚ äM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44959493 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25154750 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25144593 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2061609299456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44969652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147388898407936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191659843882816 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5666570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196009034738688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44959493 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19814900 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19814902 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19814900 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193947408399872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2055950539878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5666394 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2055863753340 trainer/global_step‚2499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18962277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚204 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145265907048960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚1671.653162612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚193947425439232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145266001158656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2061609299456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13295883 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2122991358976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19814902 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25144593 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196009034738688 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193947408399872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4@#½ï&ƒ éM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2061609299456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44969652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147388898407936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191659843882816 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5666570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196009034738688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44959493 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19814900 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19814902 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19814900 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193947408399872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2055950539878 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5666394 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2055863753340 trainer/global_step‚2499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18962277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚204 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145265907048960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚1671.653162612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚193947425439232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145266001158656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2061609299456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13295883 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2122991358976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19814902 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25144593 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196009034738688 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193947408399872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44959493 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25154750 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25144593Ì0K2„ - _wandbruntime‚1671  _runtime‚1671c’¾… € &loss/total_loss‚11.604279184452112 trainer/global_step‚2499 0loss/FRt1/assignment_loss‚1.0323441467612484 2loss/SRqqt1/assignment_loss‚0.6599571603990386 1loss/FBt2/assignment_loss‚0.12324768741131056 .loss/FRt1/detection_loss‚1.427214663647675 1loss/SRqqt2/detection_loss‚1.1370002884426726 epoch‚6 2loss/SRqqt2/assignment_loss‚0.7175994693965918 /loss/FBt1/detection_loss‚1.0620427406408655 % _timestamp‚1.7844945150527284e+09 _step‚205 1loss/FBt1/assignment_loss‚0.07815162138290235 /loss/FRt2/detection_loss‚0.5795931107999773 _runtime‚1671.654305284 /loss/FRt2/assignment_loss‚2.649236518163061 1loss/SRqqt1/detection_loss‚0.7100911526282059 /loss/FBt2/detection_loss‚1.4278006247785635+½ † … /loss/FRt2/assignment_loss‚2.649236518163061 1loss/SRqqt1/detection_loss‚0.7100911526282059 /loss/FBt2/detection_loss‚1.4278006247785635 &loss/total_loss‚11.604279184452112 trainer/global_step‚2499 0loss/FRt1/assignment_loss‚1.0323441467612484 2loss/SRqqt1/assignment_loss‚0.6599571603990386 1loss/FBt2/assignment_loss‚0.12324768741131056 .loss/FRt1/detection_loss‚1.427214663647675 1loss/SRqqt2/detection_loss‚1.1370002884426726 epoch‚6 2loss/SRqqt2/assignment_loss‚0.7175994693965918 /loss/FBt1/detection_loss‚1.0620427406408655 % _timestamp‚1.7844945150527284e+09 _step‚205 1loss/FBt1/assignment_loss‚0.07815162138290235 /loss/FRt2/detection_loss‚0.5795931107999773 _runtime‚1671.654305284Í̦!M?‡ ‚(:´ »óôÒó‚‘¿ disk.rbd0.out‚0$ disk.nvme1n1.out‚5883.8779296875 disk.md0.in‚0$ memory_percent‚96.78783416748047 cpu‚25.838142292618635 network.recv‚486793% disk./.usageGB‚1174.8248481750488 disk.dm-0.in‚0# disk.dm-0.out‚15390.33642578125 disk.dm-1.out‚1322.271484375 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚10916.09912109375& disk.nvme1c1n1.out‚5883.8779296875 network.sent‚ 3.604086e+06& disk.nvme0n1.out‚10916.09912109375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 40.42578125% proc.memory.availableMB‚ 526.28125# proc.memory.rssMB‚ 2953.37109375* proc.memory.percent‚18.025946617126465) disk./.usagePercent‚76.52335764045262 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚0 proc.cpu.threads‚17ofLj ‚(:’ »óôÒà˽¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.862' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.287333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ÐãK>‰ ‚(:³ ÊóôÒíÙˆ¿% disk.nvme1c1n1.out‚5884.572265625 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚15391.14501953125 disk.rbd0.out‚0' proc.memory.availableMB‚ 526.5234375 network.sent‚ 3.654578e+06 network.recv‚491486) disk./.usagePercent‚76.52332682967517$ disk./.usageGB‚1174.824375152588& disk.nvme0n1.out‚10917.31689453125 disk.dm-0.in‚0$ memory_percent‚96.78635597229004 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚10917.31689453125 disk.nvme0n1.in‚0 disk.md0.out‚ 40.91015625 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.025946617126465 cpu‚25.836200341452468 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚5884.572265625 disk.dm-1.out‚1322.353515625# proc.memory.rssMB‚ 2953.37109375 Œ?žŠ ‚(:“ ÊóôÒý§Ü¿ gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚119.596' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.86533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïž*¾2‹ - _wandbruntime‚1702  _runtime‚1702‡m»`ÉŒ à ' lr-AdamW/pg1‚0.00015573783384986183 ' lr-AdamW/pg2‚0.00015573783384986183 trainer/global_step‚2549 % _timestamp‚1.7844945458038359e+09 _runtime‚1702.405667804 _step‚206¨ÏÆÎ È ' lr-AdamW/pg1‚0.00015573783384986183 ' lr-AdamW/pg2‚0.00015573783384986183 trainer/global_step‚2549 % _timestamp‚1.7844945458038359e+09 _runtime‚1702.405667804 _step‚206Îsa2Ž - _wandbruntime‚1702  _runtime‚1702ŒÂ ÚO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2094405575974 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20188723 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2162886654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2100084856832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2100084856832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150179425515008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20188723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚170 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚166 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45816346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25637768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2162769794048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116860928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199729826905600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19315815 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13543627 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197629759088128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚1702.406405205 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚148016655720960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚197629742048768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197394381377212 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚197629742048768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150179636485632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 trainer/global_step‚2549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2094318789548 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197394485202998 % _timestamp‚1.7844945458047144e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19315985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195300079627024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210970624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148016749830656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199729826905600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45826491 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197629759088128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _step‚207 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199729932604928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25627625 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5772354 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_startúbY /num_device_alloc‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2100173516800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195300062587664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25637768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199729932604928 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2100173516800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20188721 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5772188 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25627625 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13543631 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150179636485632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 trainer/global_step‚2549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2094318789548 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197394485202998 % _timestamp‚1.7844945458047144e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19315985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195300079627024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210970624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148016749830656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199729826905600 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45826491 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197629759088128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _step‚207 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199729932604928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25627625 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5772354 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2100173516800 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195300062587664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25637768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199729932604928 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2100173516800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20188721 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5772188 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25627625 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13543631 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2162886654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2100084856832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2100084856832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150179425515008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20188723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚170 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚166 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45816346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25637768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2162769794048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116860928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199729826905600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19315815 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13543627 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200Ïëw³B2‘ - _wandbruntime‚1702  _runtime‚1702}jöíê&’ äM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5774374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20196352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100959239680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 % _timestamp‚1.7844945462061684e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199805867994112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150236583022592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148072887558656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2095189070686 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _step‚208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195374353113056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19323057 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45843954 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚197704908754432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197704891715072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13548683 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100870572544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199805867994112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20196352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2163578610176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25647602 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45843954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45833795 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5774556 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148072793448960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚1702.808144968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25637445 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19323243 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20196350 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195374370152416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150236372059136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197469455397204 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197704908754432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200—W–ï&“ éM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100870572544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20196350 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚186 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2100959239680 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2163695463936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199805762287616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195374353113056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19323057 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45843954 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚197704908754432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197704891715072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13548683 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100870572544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199805867994112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20196352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2163578610176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25647602 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45843954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45833795 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5774556 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148072793448960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚1702.808144968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25637445 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19323243 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20196350 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195374370152416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150236372059136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197469455397204 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197704908754432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5774374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20196352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100959239680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 % _timestamp‚1.7844945462061684e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199805867994112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150236583022592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148072887558656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2095189070686 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _step‚208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 P˜ ‚(:³ ÙóôÒ‡õŒÁ disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5885.232421875 disk.dm-0.in‚0 disk.dm-1.out‚ 1322.52734375 proc.cpu.threads‚17 disk.nvme0n1.in‚0& disk.nvme0n1.out‚10919.31494140625 disk.md0.in‚0# disk.dm-0.out‚15393.04736328125( proc.memory.availableMB‚ 526.20703125# proc.memory.rssMB‚ 2953.37109375 cpu‚25.83408145926901 disk.nvme1n1.in‚0 disk.md0.out‚ 41.17578125 network.sent‚ 3.657542e+06% disk./.usageGB‚1174.8245544433594( disk.nvme0c0n1.out‚10919.31494140625 disk.dm-1.in‚0$ memory_percent‚96.78828716278076# disk.nvme1n1.out‚5885.232421875 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.025946617126465 network.recv‚494985) disk./.usagePercent‚76.52333850795371 disk.nvme0c0n1.in‚0&FÙ?™ ‚(:´ èóôÒè ý¾( disk.nvme0c0n1.out‚10920.24560546875 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚5885.7998046875 disk.dm-0.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚5885.7998046875$ memory_percent‚96.78833484649658( proc.memory.availableMB‚ 526.19921875* proc.memory.percent‚18.025946617126465 cpu‚25.83169684969364 proc.cpu.threads‚17) disk./.usagePercent‚76.52333975032377% disk./.usageGB‚1174.8245735168457# disk.dm-0.out‚15393.68798828125 disk.md0.out‚ 41.52734375# proc.memory.rssMB‚ 2953.37109375 network.sent‚ 3.721657e+06 network.recv‚498861 disk.rbd0.out‚0& disk.nvme0n1.out‚10920.24560546875 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 1322.6171875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0s£¸½“š ‚(:ˆ èóôҧó¿ gpu.0.gpu‚92.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.104' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b˜Æd2› - _wandbruntime‚1733  _runtime‚1733m;ûÉœ à ' lr-AdamW/pg1‚0.00015551829999721934 ' lr-AdamW/pg2‚0.00015551829999721934 trainer/global_step‚2599 % _timestamp‚1.7844945772845898e+09 _runtime‚1733.886440084 _step‚210*³Î È ' lr-AdamW/pg1‚0.00015551829999721934 ' lr-AdamW/pg2‚0.00015551829999721934 trainer/global_step‚2599 % _timestamp‚1.7844945772845898e+09 _runtime‚1733.886440084 _step‚210Òþm»2ž - _wandbruntime‚1733  _runtime‚1733®¢nÈà'Ÿ ÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20570173 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201387225363968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153026858549760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2203445872640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20570171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201148146177620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203526660161024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26120477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 _step‚211 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203526765860352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5880668 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2139434797056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201387225363968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 % _timestamp‚1.7844945772855854e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26130620 _runtime‚1733.887389825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚178 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5880842 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46690648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19677273 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201148250003406 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2139523457024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46700793 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20570173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20570171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2133557320356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2139434797056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150823542120960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2133644106782 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13796431 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19677095 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2139523457024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203526660161024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150823636230656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26120477 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153027082103296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203526765860352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13796427 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201387242403328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360´_‚¡å'  ßO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153027082103296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203526765860352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13796427 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201387242403328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20570173 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201387225363968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153026858549760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2203445872640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20570171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201148146177620 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203526660161024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26120477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 _step‚211 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203526765860352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5880668 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2139434797056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201387225363968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 % _timestamp‚1.7844945772855854e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26130620 _runtime‚1733.887389825 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚178 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5880842 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46690648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19677273 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201148250003406 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2139523457024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46700793 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20570173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20570171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2133557320356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2139434797056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150823542120960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2133644106782 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13796431 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19677095 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2139523457024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203526660161024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150823636230656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26120477 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Óíüq 2¡ - _wandbruntime‚1734  _runtime‚1734 X#…¢ äM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46708097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5882880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153083806535680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150879773958656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201462375030272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26130297 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2140309179904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203602701249536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20577800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5882694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2134340814956 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26140454 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2204256123392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1734.290508421 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201462375030272 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20577802 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19684177 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26130297 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199088896422016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46718256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13801483 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13801487 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46708097 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203602701249536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150879679848960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2204126686720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ›Þûï&£ éM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46708097 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203602701249536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150879679848960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2204126686720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46708097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5882880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153083806535680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150879773958656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201462375030272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26130297 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2140309179904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203602701249536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20577800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5882694 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2134340814956 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26140454 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2204256123392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1734.290508421 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201462375030272 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20577802 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19684177 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26130297 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199088896422016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46718256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13801483 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13801487 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350ÔqeÚÒ2¤ - _wandbruntime‚1734  _runtime‚1734'>ûØ¥ þ /loss/FBt1/detection_loss‚1.1129989203796724 epoch‚6 _runtime‚1734.291012532 2loss/SRqqt1/assignment_loss‚0.6870778897676953 0loss/FBt2/assignment_loss‚0.2962579953012387 0loss/SRqqt2/detection_loss‚1.072537377713469 .loss/FBt2/detection_loss‚3.699527803241173 % _timestamp‚1.7844945776894841e+09 _step‚213 0loss/FRt1/assignment_loss‚1.0083924034650331 0loss/FRt2/assignment_loss‚2.6208533299151466 2loss/SRqqt2/assignment_loss‚0.7435455605782316 2loss/FBt1/assignment_loss‚0.048705555448559375 &loss/total_loss‚14.048981134029976 trainer/global_step‚2599 /loss/FRt2/detection_loss‚0.5853883697121103 -loss/FRt1/detection_loss‚1.46607836516466 1loss/SRqqt1/detection_loss‚0.7076175633429881toC ¦ ƒ -loss/FRt1/detection_loss‚1.46607836516466 1loss/SRqqt1/detection_loss‚0.7076175633429881 /loss/FBt1/detection_loss‚1.1129989203796724 epoch‚6 _runtime‚1734.291012532 2loss/SRqqt1/assignment_loss‚0.6870778897676953 0loss/FBt2/assignment_loss‚0.2962579953012387 0loss/SRqqt2/detection_loss‚1.072537377713469 .loss/FBt2/detection_loss‚3.699527803241173 % _timestamp‚1.7844945776894841e+09 _step‚213 0loss/FRt1/assignment_loss‚1.0083924034650331 0loss/FRt2/assignment_loss‚2.6208533299151466 2loss/SRqqt2/assignment_loss‚0.7435455605782316 2loss/FBt1/assignment_loss‚0.048705555448559375 &loss/total_loss‚14.048981134029976 trainer/global_step‚2599 /loss/FRt2/detection_loss‚0.5853883697121103Õ:ßÕðA§ ‚(:¶ ÷óôÒÔ—¿ disk.rbd0.out‚0 disk.nvme0c0n1.in‚0* proc.memory.percent‚18.025946617126465# disk.dm-0.out‚15400.16650390625 disk.rbd0.in‚0& disk.nvme0n1.out‚10926.96142578125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚5886.7880859375$ memory_percent‚96.78854942321777' proc.memory.availableMB‚ 526.1640625 network.sent‚ 3.724579e+06 network.recv‚502595! disk.dm-1.out‚1322.7353515625 disk.md0.in‚0 disk.md0.out‚ 42.05078125# proc.memory.rssMB‚ 2953.37109375 cpu‚25.829714601638127 proc.cpu.threads‚17) disk./.usagePercent‚76.52333353847348$ disk./.usageGB‚1174.824478149414 disk.dm-1.in‚0( disk.nvme0c0n1.out‚10926.96142578125 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚5886.7880859375 disk.dm-0.in‚0Ûù÷<“¨ ‚(:ˆ ÷óôÒڇܿ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚221.529' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð¬´G=© ‚(:² †ôôÒ ¼î¾ network.sent‚ 3.787928e+06( disk./.usagePercent‚76.5233424835379 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚10927.98779296875 disk.md0.out‚42.46875 network.recv‚506576 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0# disk.dm-0.out‚15400.82666015625$ memory_percent‚96.78857326507568( proc.memory.availableMB‚ 526.16015625! disk.dm-1.out‚1322.7900390625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0# disk.nvme1n1.out‚5887.380859375 cpu‚25.827770896425502% disk./.usageGB‚1174.8246154785156& disk.nvme0n1.out‚10927.98779296875# proc.memory.rssMB‚ 2953.37109375* proc.memory.percent‚18.025946617126465 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5887.380859375 Œ¨Ïžª ‚(:“ †ôôÒÞ¶±¿ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚225.854' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.28466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$±c‹2« - _wandbruntime‚1765  _runtime‚1765w%¤qɬ à ' lr-AdamW/pg1‚0.00015529393020426177 ' lr-AdamW/pg2‚0.00015529393020426177 trainer/global_step‚2649 % _timestamp‚1.7844946088171017e+09 _runtime‚1765.420257528 _step‚214#°ê]έ È ' lr-AdamW/pg1‚0.00015529393020426177 ' lr-AdamW/pg2‚0.00015529393020426177 trainer/global_step‚2649 % _timestamp‚1.7844946088171017e+09 _runtime‚1765.420257528 _step‚214Ö¸(ÈT2® - _wandbruntime‚1765  _runtime‚1765k&|à'¯ ÚO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202729132166224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202729115126864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚177 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205144725718528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5987585 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207323493416448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2244003395072 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20036812 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47575095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5987762 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207323493416448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47564950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26613329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153630428520960 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2172795851164 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚205144708679168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205144725718528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584À»*z° ßO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205144708679168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155874526025728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2172882637590 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20951623 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207323493416448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2244003395072 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20036812 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47575095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5987762 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207323493416448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47564950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26613329 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153630428520960 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2172795851164 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚205144708679168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205144725718528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202729132166224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202729115126864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚177 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205144725718528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5987585 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20959250 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207399534504960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47592558 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2179659120128 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5989685 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2173666132302 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚164 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2179659120128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204977088823918 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202803405652256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207399428798464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204976984998020 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47592558 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207399534504960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47582399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2179570452992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47582399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26633306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2649 % _timestamp‚1.7844946092201865e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205219875384832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26623149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155931483572736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155931264220672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2179570452992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26623149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153686660358656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20043968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚1765.822173567 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14054283 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5989849 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205219875384832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205219858345472 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚164 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2179659120128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204977088823918 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202803405652256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207399428798464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204976984998020 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47592558 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207399534504960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47582399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2179570452992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47582399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26633306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2649 % _timestamp‚1.7844946092201865e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205219875384832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26623149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155931483572736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155931264220672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2179570452992 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26623149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153686660358656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20043968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚1765.822173567 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14054283 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5989849 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205219875384832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205219858345472 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20959250 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207399534504960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ؼÁQ¨2´ - _wandbruntime‚1765  _runtime‚1765Kö:µ  .loss/FBt2/detection_loss‚1.983309242860425 1loss/FBt1/assignment_loss‚0.09372882960424582 /loss/FRt1/detection_loss‚1.5129349264124718 /loss/FRt2/detection_loss‚0.5786020191689618 &loss/total_loss‚12.369720198322185 epoch‚6 trainer/global_step‚2649 % _timestamp‚1.7844946092211273e+09 2loss/SRqqt2/assignment_loss‚0.7891369811235849 1loss/FBt2/assignment_loss‚0.17910587344159246 0loss/FRt2/assignment_loss‚2.6209753032300616 1loss/SRqqt2/detection_loss‚1.1463563066851161 _runtime‚1765.822616037 _step‚217 0loss/FRt1/assignment_loss‚1.1473986113093235 2loss/SRqqt1/assignment_loss‚0.7170629990552783 1loss/SRqqt1/detection_loss‚0.6949447819423562 /loss/FBt1/detection_loss‚0.9061643234887669¥†ò ¶ † % _timestamp‚1.7844946092211273e+09 2loss/SRqqt2/assignment_loss‚0.7891369811235849 1loss/FBt2/assignment_loss‚0.17910587344159246 0loss/FRt2/assignment_loss‚2.6209753032300616 1loss/SRqqt2/detection_loss‚1.1463563066851161 _runtime‚1765.822616037 _step‚217 0loss/FRt1/assignment_loss‚1.1473986113093235 2loss/SRqqt1/assignment_loss‚0.7170629990552783 1loss/SRqqt1/detection_loss‚0.6949447819423562 /loss/FBt1/detection_loss‚0.9061643234887669 .loss/FBt2/detection_loss‚1.983309242860425 1loss/FBt1/assignment_loss‚0.09372882960424582 /loss/FRt1/detection_loss‚1.5129349264124718 /loss/FRt2/detection_loss‚0.5786020191689618 &loss/total_loss‚12.369720198322185 epoch‚6 trainer/global_step‚2649Ù*8oA· ‚(:¶ •ôôÒŽÅ–¿ disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 network.sent‚ 3.805632e+06 disk.rbd0.out‚0& disk.nvme0n1.out‚10930.01123046875 disk.nvme1n1.in‚0 disk.dm-0.in‚0# memory_percent‚96.7888355255127' proc.memory.availableMB‚ 526.1171875 cpu‚25.825705999252307 network.recv‚510317% disk./.usageGB‚1173.6139831542969 disk.nvme0c0n1.in‚0% disk.nvme1n1.out‚5888.03662109375 disk.md0.out‚ 42.796875# disk.dm-0.out‚15402.69873046875" disk.dm-1.out‚1322.89599609375 proc.cpu.threads‚17( disk./.usagePercent‚76.4444867711641 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10930.01123046875' disk.nvme1c1n1.out‚5888.03662109375 disk.md0.in‚0# proc.memory.rssMB‚ 2953.37109375* proc.memory.percent‚18.025946617126465 disk.nvme0n1.in‚0¨Ë¿ž¸ ‚(:“ •ôôÒ¡ÖÇ¿ gpu.0.gpu‚91.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚126.769' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.25633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ˜wD¹ ‚(:¹ ¤ôôÒÑÚ÷¾ disk.dm-0.in‚0# disk.dm-0.out‚15403.51904296875" disk.dm-1.out‚1323.06005859375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10931.09130859375% disk.nvme1n1.out‚5888.67138671875' proc.memory.availableMB‚ 526.3671875 proc.cpu.threads‚17 network.sent‚ 3.853166e+06) disk./.usagePercent‚76.44452205447378 disk.md0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚10931.09130859375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5888.67138671875 disk.nvme1n1.in‚0 network.recv‚513927 disk.md0.out‚ 43.1328125 disk.dm-1.in‚0$ memory_percent‚96.78730964660645# proc.memory.rssMB‚ 2953.37109375* proc.memory.percent‚18.025946617126465% disk./.usageGB‚1173.6145248413086 disk.rbd0.out‚0 disk.nvme0n1.in‚0 cpu‚25.823692252421065e+Z‚Ÿº ‚(:” ¤ôôÒÉȹ¿ gpu.0.gpu‚67.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚170.854' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚56.951333333333345 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èâñî2» - _wandbruntime‚1796  _runtime‚1796Ùá²}Ǽ Á trainer/global_step‚2699 % _timestamp‚1.7844946399261901e+09 _runtime‚1796.528497524 _step‚218 & lr-AdamW/pg1‚0.0001550647388818417 & lr-AdamW/pg2‚0.0001550647388818417â©e̽ Æ & lr-AdamW/pg1‚0.0001550647388818417 & lr-AdamW/pg2‚0.0001550647388818417 trainer/global_step‚2699 % _timestamp‚1.7844946399261901e+09 _runtime‚1796.528497524 _step‚218ÚT{Z12¾ - _wandbruntime‚1796  _runtime‚1796tÄÄ¿ ÚO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218223337472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27106181 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156437314920960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158721991682048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14302031 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48449397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6094112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚219 ÃëÊhÄ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218223337472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211120326671872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158721768128512 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27116324 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚208902209033728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218134677504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206443641396464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21333073 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6094291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156437409030656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2212121168398 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚2699 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211120326671872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚208902191994368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211120432371200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 _runtime‚1796.528710254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20396322 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2284582651392 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208902191994368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218134677504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚208902209033728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2284453207552 % _timestamp‚1.7844946399268484e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211120326671872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158721768128512 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27116324 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚208902209033728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218134677504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206443641396464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21333073 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6094291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156437409030656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2212121168398 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚2699 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211120326671872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚208902191994368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211120432371200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 _runtime‚1796.528710254 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20396322 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2284582651392 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208902191994368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218134677504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚208902209033728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2284453207552 % _timestamp‚1.7844946399268484e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48449397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6094112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚219Û@Î ÷2Á - _wandbruntime‚1796  _runtime‚1796Lm ê& äM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48466860 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 _runtime‚1796.930626645 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158778719759872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48456701 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚208977341660672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2218920393216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156493546758656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚206517931921856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2212904663110 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211196367760384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218920393216 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 % _timestamp‚1.7844946403286438e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211196367760384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14307087 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2285388158976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208977358700032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6096263 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20403346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208730853624326 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158778934917632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156493452648960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21340700 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21340700 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 _step‚220 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21340702 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27116001 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211196262053888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2285267110912 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48456701 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682(‘È2à éM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20403346 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208730853624326 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158778934917632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156493452648960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21340700 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21340700 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 _step‚220 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21340702 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27116001 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211196262053888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2285267110912 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48456701 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48466860 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 _runtime‚1796.930626645 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158778719759872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48456701 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚208977341660672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2218920393216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156493546758656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚206517931921856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2212904663110 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211196367760384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218920393216 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 % _timestamp‚1.7844946403286438e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211196367760384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14307087 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2285388158976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208977358700032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6096263 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20755924 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159244295430656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159244201320960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844946713521643e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21714521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210158184705424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210158167666064 _step‚223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2257484617728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2257573277696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6201093 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251359699206 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14554827 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212409544404630 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212659675309568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20755735 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212409440578844 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚214917265626624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49313554 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2257484617728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21714521 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161569196936704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚214917265626624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27609176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27599033 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49323699 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27609176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212659692348928 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚214917159927296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161569414198784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 trainer/global_step‚2749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27599033 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4/kò$å'Ð ßO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21714521 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210158184705424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210158167666064 _step‚223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2257484617728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2257573277696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚189 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6201093 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251359699206 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14554827 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212409544404630 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212659675309568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20755735 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212409440578844 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚214917265626624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49313554 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2257484617728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21714521 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161569196936704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚214917265626624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27609176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27599033 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49323699 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27609176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212659692348928 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚214917159927296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161569414198784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 trainer/global_step‚2749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27599033 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20755924 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159244295430656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159244201320960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844946713521643e+09ß~y+û2Ñ - _wandbruntime‚1828  _runtime‚1828tÍ$‚Û Ò äM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49341162 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159300433158656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212484514598836 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21722150 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161626362439680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212734824975872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21722150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210232458191456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27619010 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2258270333440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27608853 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2258359000576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚214993201015808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large殎E_pool.allocated‚210232475230816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2252143193918 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844946717544653e+09 _runtime‚1828.356414449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212734824975872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2258270333440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27608853 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2252056407380 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161626138893312 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2325799844352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49331003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14559887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6203186 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20763073 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159300339048960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21722152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20762891 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49331003 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27619010 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49341162 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚2749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2325929281024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214993201015808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212484618424734 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212734842015232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21722152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20762891 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49331003 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27619010 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49341162 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚2749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2325929281024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214993201015808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212484618424734 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212734842015232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212484514598836 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21722150 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161626362439680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212734824975872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21722150 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812àâà€$2Ô - _wandbruntime‚1828  _runtime‚1828þ”?IÕ  1loss/FBt1/assignment_loss‚0.13218232823091258 /loss/FBt1/detection_loss‚0.8076444826645081 /loss/FRt1/detection_loss‚1.5417593794224076 .loss/FRt2/detection_loss‚0.605590902834687 &loss/total_loss‚11.051902676221104 _step‚225 0loss/FRt2/assignment_loss‚2.5605666201112016 1loss/SRqqt1/detection_loss‚0.6956830755558042 /loss/FBt2/detection_loss‚0.8174425380798183 trainer/global_step‚2749 _runtime‚1828.356925079 0loss/FRt1/assignment_loss‚1.1467066380907507 2loss/SRqqt1/assignment_loss‚0.7279890983270791 2loss/SRqqt2/assignment_loss‚0.7282543054975609 1loss/FBt2/assignment_loss‚0.13396126421326848 1loss/SRqqt2/detection_loss‚1.1541220431931059 epoch‚6 % _timestamp‚1.7844946717553825e+0956 y Ö † 1loss/FBt1/assignment_loss‚0.13218232823091258 /loss/FBt1/detection_loss‚0.8076444826645081 /loss/FRt1/detection_loss‚1.5417593794224076 .loss/FRt2/detection_loss‚0.605590902834687 &loss/total_loss‚11.051902676221104 _step‚225 0loss/FRt2/assignment_loss‚2.5605666201112016 1loss/SRqqt1/detection_loss‚0.6956830755558042 /loss/FBt2/detection_loss‚0.8174425380798183 trainer/global_step‚2749 _runtime‚1828.356925079 0loss/FRt1/assignment_loss‚1.1467066380907507 2loss/SRqqt1/assignment_loss‚0.7279890983270791 2loss/SRqqt2/assignment_loss‚0.7282543054975609 1loss/FBt2/assignment_loss‚0.13396126421326848 1loss/SRqqt2/detection_loss‚1.1541220431931059 epoch‚6 % _timestamp‚1.7844946717553825e+09á|yZ=× ‚(:² ÑôôÒ‚’§¿ disk.md0.out‚ 44.3046875 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 cpu‚25.817951030674273 network.recv‚524930 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚10946.14599609375' proc.memory.availableMB‚ 526.2265625# proc.memory.rssMB‚ 2953.37109375$ disk.nvme1n1.out‚5894.9658203125 disk.dm-0.out‚ 15418.7890625& disk.nvme0n1.out‚10946.14599609375$ memory_percent‚96.78816795349121* proc.memory.percent‚18.025946617126465 proc.cpu.threads‚17 network.sent‚ 3.939042e+06$ disk./.usageGB‚1173.668960571289 disk.nvme1n1.in‚0" disk.dm-1.out‚1326.61669921875 disk.rbd0.in‚0& disk.nvme1c1n1.out‚5894.9658203125) disk./.usagePercent‚76.44806777862166߀ÙØ ‚(:’ ÑôôÒ°ÐÙ¿ gpu.0.gpu‚58.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.097' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.36566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø Jü:Ù ‚(:¯ àôôÒ¡é—¿ network.sent‚ 3.989382e+06( disk./.usagePercent‚76.4365057859062' disk.nvme0c0n1.out‚10947.6962890625 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚5895.50830078125 disk.md0.in‚0" disk.dm-1.out‚1326.70263671875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚5895.50830078125 disk.dm-0.in‚0$ memory_percent‚96.78888320922852& proc.memory.availableMB‚ 526.109375# proc.memory.rssMB‚ 2953.37109375* proc.memory.percent‚18.025946617126465 network.recv‚529105% disk.nvme0n1.out‚10947.6962890625 disk.md0.out‚ 44.578125 disk.dm-1.in‚0 cpu‚25.816195990380656$ disk./.usageGB‚1173.491455078125 disk.nvme1n1.in‚0 disk.dm-0.out‚ 15420.1796875 disk.rbd0.in‚0 proc.cpu.threads‚17¶ïMhžÚ ‚(:“ àôôÒ—à̾ gpu.0.gpu‚91.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚222.967' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.32233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HWË«2Û - _wandbruntime‚1859  _runtime‚1859~ß–ˆÉÜ Ã ' lr-AdamW/pg2‚0.00015459195083946783 trainer/global_step‚2799 % _timestamp‚1.7844947024472816e+09 _runtime‚1859.049121185 _step‚226 ' lr-AdamW/pg1‚0.00015459195083946783‚ª©wÎÝ È % _timestamp‚1.7844947024472816e+09 _runtime‚1859.049121185 _step‚226 ' lr-AdamW/pg1‚0.00015459195083946783 ' lr-AdamW/pg2‚0.00015459195083946783 trainer/global_step‚2799âÔÎ`t2Þ - _wandbruntime‚1859  _runtime‚1859}»ß´à'ß ÚO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164416775161344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2296923217920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚218713993182720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14807627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚213872693935664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 trainer/global_step‚2799 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2296834557952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21115244 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2365593330688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22095973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28091885 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚216417175664128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚216417158624768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2365459692544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216163309205038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2290598230014 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218713993182720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218714098882048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚216417158624768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2296834557952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22095971 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28102028 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21115055üEj°Çà ßO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164416775161344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2296923217920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚218713993182720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14807627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚213872693935664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 trainer/global_step‚2799 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2296834557952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21115244 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2365593330688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22095973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_po¢%ƒ7"ol.current‚88659968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28091885 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚216417175664128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚216417158624768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2365459692544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216163309205038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚185 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2290598230014 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218713993182720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218714098882048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚216417158624768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2296834557952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22095971 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28102028 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21115055ã÷\½2á - _wandbruntime‚1859  _runtime‚1859qW"4ê&â äM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚216492325330432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2291294938188 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50215464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50205305 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164473716037632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162107225448960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218790034271232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28111862 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6309592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2366267042304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213946984461056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2297708940800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22103602 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28111862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _step‚228 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6309777 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216238279399244 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2366396478976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21122275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50205305 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22103602 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218790034271232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28101705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14812687 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2297708940800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22103600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2799 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 _runtime‚1859.450156205 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚216492308291072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14812683 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164473492491264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28101705 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162107319558656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2297620273664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218789928564736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22103602 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218790034271232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28101705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14812687 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2297708940800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22103600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2799 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 _runtime‚1859.450156205 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚216492308291072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14812683 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164473492491264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28101705 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162107319558656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2297620273664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218789928564736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚216492325330432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2291294938188 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50215464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50205305 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164473716037632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162107225448960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218790034271232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28111862 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6309592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2366267042304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213946984461056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2297708940800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22103602 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28111862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _step‚228 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6309777 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216238279399244 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2366396478976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21122275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50205305 Eç ‚(:³ ïôôÒ¿¨•¿% disk./.usageGB‚1172.2294311523438 disk.nvme1n1.in‚0# disk.nvme1n1.out‚5898.595703125 disk.md0.out‚ 44.890625$ memory_percent‚96.79062366485596( proc.memory.availableMB‚ 525.82421875# proc.memory.rssMB‚ 2953.37109375* proc.memory.percent‚18.025946617126465) disk./.usagePercent‚76.35430263164591 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5898.595703125 disk.md0.in‚0 cpu‚25.814332381618954 proc.cpu.threads‚17 network.sent‚ 4.005147e+06 network.recv‚532495 disk.rbd0.in‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚10963.68505859375 disk.dm-0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚1326.99462890625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚10963.68505859375! disk.dm-0.out‚15438.294921875ø;T‡œè ‚(:‘ ïôôÒö¨Á¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚114.698' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.23266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06G‰}2é - _wandbruntime‚1873  _runtime‚1873¯N£Þ7ê ®{ 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15193870663642883 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7354372143745422 ,jet/accuracy_4_of_4‚0.011098379269242287 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6946824193000793 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611111044883728 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7526367902755737 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6892315745353699 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5835867524147034 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6462672352790833 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06749028712511063 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0654914528131485 (particle/accuracy‚0.8461333513259888 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07155586779117584 (p] }9"Purity/0FRt0SRqqt0FBt/event_purity‚1 _runtime‚1873.401416873 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6952740550041199 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6351870894432068 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24171733856201172 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08725916594266891 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9864369630813599 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1756000816822052 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23915790021419525 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5453886389732361 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2062317579984665 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9948822259902954 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 /particle/accuracy_3_of_4‚0.9917800426483154 particle/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9605923891067505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.974976122379303 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6409479379653931 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5824741721153259 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19752484560012817 +particle/specificity‚0.9112271666526794 % _timestamp‚1.7844947167991116e+09 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.571031928062439 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04755594953894615 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24034695327281952 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6081045269966125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20526504516601562 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 *jet/accuracy_2_of_4‚0.8104543089866638 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13710054755210876 particle/accuracy_4_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.251517117023468 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6930596828460693 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6900504231452942 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6354552507400513 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7999622821807861 7validation_average_jet_accuracy‚0.37686190009117126 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.657155454158783 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07370255142450333 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3824082016944885 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8004534840583801 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7993900775909424 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.985820472240448 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.776750385761261 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8270481824874878 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892680048942566 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8857359886169434 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6408179402351379 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6091283559799194 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959607720375061 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4041033685207367 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7476677894592285 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03765074536204338 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5387499928474426 jet/accuracy_4_of_5‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.994054913520813 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6993197202682495 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8518518209457397 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9945683479309082 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1733333319425583 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5494129061698914 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5455774068832397 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.027165640145540237 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39284273982048035 *loss/val_total_loss‚10.729960341003869 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8084089159965515 trainer/global_step‚2799 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29977479577064514 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9833170771598816 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9342737793922424 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44900432229042053 *jet/accuracy_1_of_4‚0.9737879633903503 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27129843831062317 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_4_of_4‚0.9502105712890625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917820692062378 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13999740779399872 *jet/accuracy_1_of_3‚0.9268994927406311 +jet/accuracy_3_of_3‚0.06703018397092819 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6356059312820435 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9671369194984436 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7527335286140442 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3802083432674408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6090598106384277 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291865348816 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8820939660072327 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24171733856201172 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48027393221855164 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6462672352790833 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7302422523498535 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07244522124528885 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 /particle/accuracy_3_of_3‚0.9990025758743286 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3257247507572174 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24007050693035126 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29204005002975464 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1835794895887375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12084716558456421 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6618538498878479 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9844000935554504 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9679276347160339 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38205060362815857 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6867435574531555 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25162872672080994 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8251000642776489 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2335543930530548 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 6"Purity/0FRt*SRqqt2FBt/event_purity‚0.8793905377388 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5553610920906067 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7168366312980652 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8128117918968201 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19259493052959442 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21353428065776825 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05179769545793533 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_5_of_5‚0 epoch‚6 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 particle/accuracy_5_of_5‚0 _step‚230 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888485670089722 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6199892163276672 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7653628587722778 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7578131556510925 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2176949828863144 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21723365783691406 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7791380882263184 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9955384135246277 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22089837491512299 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6129286885261536 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7989690899848938 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9945683479309082 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07083303481340408 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9860723614692688 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47484925389289856 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6067930459976196 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0654914528131485 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8270481824874878 'particle/f_score‚0.6292757391929626 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38132140040397644 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.837951123714447 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33662623167037964 *jet/accuracy_1_of_2‚0.7990585565567017 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1355302631855011 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17884349822998047 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 particle/accuracy_3_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22438156604766846 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.995842695236206 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6207618713378906 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03513747826218605 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4728219211101532 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2524471580982208ÿN.¹=ë ³{ & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1733333319425583 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5494129061698914 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5455774068832397 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.027165640145540237 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.39284273982048035 *loss/val_total_loss‚10.729960341003869 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8084089159965515 trainer/global_step‚2799 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29977479577064514 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9833170771598816 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9342737793922424 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44900432229042053 *jet/accuracy_1_of_4‚0.9737879633903503 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27129843831062317 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_4_of_4‚0.9502105712890625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917820692062378 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13999740779399872 *jet/accuracy_1_of_3‚0.9268994927406311 +jet/accuracy_3_of_3‚0.06703018397092819 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6356059312820435 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9671369194984436 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7527335286140442 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3802083432674408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6090598106384277 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291865348816 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8820939660072327 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24171733856201172 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48027393221855164 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6462672352790833 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7302422523498535 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07244522124528885 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 /particle/accuracy_3_of_3‚0.9990025758743286 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3257247507572174 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24007050693035126 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29204005002975464 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1835794895887375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12084716558456421 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6618538498878479 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9844000935554504 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9679276347160339 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38205060362815857 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6867435574531555 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25162872672080994 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8251000642776489 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2335543930530548 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 6"Purity/0FRt*SRqqt2FBt/event_purity‚0.8793905377388 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5553610920906067 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7168366312980652 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8128117918968201 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19259493052959442 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21353428065776825 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05179769545793533 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_5_of_5‚0 epoch‚6 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 particle/accuracy_5_of_5‚0 _step‚230 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888485670089722 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6199892163276672 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7653628587722778 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7578131556510925 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2176949828863144 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21723365783691406 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7791380882263184 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9955384135246277 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22089837491512299 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6129286885261536 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7989690899848938 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9945683479309082 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07083303481340408 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9860723614692688 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47484925389289856 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6067930459976196 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0654914528131485 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8270481824874878 'particle/f_score‚0.6292757391929626 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38132140040397644 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.837951123714447 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33662623167037964 *jet/accuracy_1_of_2‚0.7990585565567017 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1355302631855011 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17884349822998047 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 particle/accuracy_3_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22438156604766846 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.995842695236206 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6207618713378906 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03513747826218605 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4728219211101532 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2524471580982208 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15193870663642883 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7354372143745422 ,jet/accuracy_4_of_4‚0.011098379269242287 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6946824193000793 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611111044883728 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7526367902755737 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6892315745353699 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5835867524147034 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6462672352790833 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06749028712511063 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0654914528131485 (particle/accuracy‚0.8461333513259888 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07155586779117584 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 _runtime‚1873.401416873 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6952740550041199 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6351870894432068 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24171733856201172 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08725916594266891 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9864369630813599 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1756000816822052 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23915790021419525 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5453886389732361 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2062317579984665 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9948822259902954 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 /particle/accuracy_3_of_4‚0.9917800426483154 particle/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9605923891067505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.974976122379303 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6409479379653931 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5824741721153259 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19752484560012817 +particle/specificity‚0.9112271666526794 % _timestamp‚1.7844947167991116e+09 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.571031928062439 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04755594953894615 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24034695327281952 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6081045269966125 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20526504516601562 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 *jet/accuracy_2_of_4‚0.8104543089866638 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13710054755210876 particle/accuracy_4_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.251517117023468 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6930596828460693 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6900504231452942 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6354552507400513 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7999622821807861 7validation_average_jet_accuracy‚0.37686190009117126 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.657155454158783 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07370255142450333 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3824082016944885 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8004534840583801 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7993900775909424 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.985820472240448 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.776750385761261 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8270481824874878 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892680048942566 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8857359886169434 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6408179402351379 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6091283559799194 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959607720375061 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4041033685207367 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7476677894592285 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03765074536204338 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5387499928474426 jet/accuracy_4_of_5‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.994054913520813 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6993197202682495 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8518518209457397 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9945683479309082 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068æÉgi:ì ‚(:¯ þôôÒ¥½Ó¾# proc.memory.rssMB‚ 2957.33203125 network.sent‚ 4.054977e+06$ disk./.usageGB‚1172.229507446289) disk./.usagePercent‚76.35430760112614' disk.nvme1c1n1.out‚5962.90771484375 disk.md0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚10964.6728515625* proc.memory.percent‚18.050122261047363 cpu‚25.83567436416256 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚5962.90771484375 disk.dm-0.in‚0! disk.dm-0.out‚15439.068359375 disk.dm-1.in‚0" disk.dm-1.out‚1390.94384765625' disk.nvme0c0n1.out‚10964.6728515625 network.recv‚536659 disk.md0.out‚45.15625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0# memory_percent‚97.2081184387207& proc.memory.availableMB‚ 457.421875ëªi±Ÿí ‚(:” þôôÒÓÿª¿ gpu.0.gpu‚93.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚187.184' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚62.394666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fçã@î ‚(:µ õôÒ’•â¾ disk.nvme0n1.in‚0# disk.dm-0.out‚15442.16552734375! disk.dm-1.out‚1391.0654296875* proc.memory.percent‚18.050122261047363 cpu‚25.83342039942502% disk./.usageGB‚1172.2296028137207 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚5963.916015625 disk.md0.out‚ 45.75390625$ memory_percent‚97.20685482025146( proc.memory.availableMB‚ 457.62890625# proc.memory.rssMB‚ 2957.33203125 network.sent‚ 4.113723e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.in‚0& disk.nvme0n1.out‚10968.14892578125 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5963.916015625 disk.dm-0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚540661) disk./.usagePercent‚76.35431381297644( disk.nvme0c0n1.out‚10968.14892578125î©£‡ï ‚(:‘ õôÒ—»¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocated¯°ÅÂBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚220.452' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.48400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ÒW„>ð ‚(:³ œõôÒíŒâ¾ cpu‚25.83107468147822) disk./.usagePercent‚76.33891215135718 disk.nvme0c0n1.in‚0 disk.md0.out‚ 46.015625 disk.dm-1.in‚0* proc.memory.percent‚18.050122261047363 proc.cpu.threads‚17 network.sent‚ 4.116645e+06 network.recv‚543762 disk.nvme0n1.in‚0% disk.nvme0n1.out‚10973.9169921875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚6028.31689453125 disk.dm-0.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚1171.993148803711 disk.md0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2957.33203125' disk.nvme0c0n1.out‚10973.9169921875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚6028.31689453125# disk.dm-0.out‚15447.77099609375! disk.dm-1.out‚1455.0458984375$ memory_percent‚97.20578193664551' proc.memory.availableMB‚ 457.8046875àñ ‚(:’ œõôÒˆ…£¿ gpu.0.gpu‚32.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.622' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.20733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 °„w2ò - _wandbruntime‚1907  _runtime‚1907€4„Éó à ' lr-AdamW/pg1‚0.00015434838448580648 ' lr-AdamW/pg2‚0.00015434838448580648 trainer/global_step‚2849 % _timestamp‚1.7844947504271662e+09 _runtime‚1907.028984285 _step‚231h»Îô È _runtime‚1907.028984285 _step‚231 ' lr-AdamW/pg1‚0.00015434838448580648 ' lr-AdamW/pg2‚0.00015434838448580648 trainer/global_step‚2849 % _timestamp‚1.7844947504271662e+09çòqîm2õ - _wandbruntime‚1907  _runtime‚1907b–úÑà'ö ÚO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2351613875200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351702535168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2421762575360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223522848810496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15168686 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218560704280816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2345165232962 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223522848810496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51276610 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15168682 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844947504281871e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221171129235968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218560687241456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221171146275328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168083196011008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168082982943232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2849 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118958080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220905852474418 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599221 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599221 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21631955 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220905956300204 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223522743111168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2351613875200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6463269 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6463091 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221171146275328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28667244 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚1907.030036977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221171129235968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21631773 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51266465 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28677387 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22599223 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¹2¡5å'÷ ßO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221171129235968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218560687241456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221171146275328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168083196011008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168082982943232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2849 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118958080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220905852474418 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599221 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599221 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21631955 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220905956300204 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223522743111168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2351613875200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6463269 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6463091 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221171146275328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28667244 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚1907.030036977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221171129235968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21631773 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51266465 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28677387 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22599223 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2351613875200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351702535168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2421762575360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223522848810496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15168686 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218560704280816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2345165232962 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223522848810496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51276610 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15168682 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844947504281871e+09èp…¢2ø - _wandbruntime‚1907  _runtime‚1907$¸é&ù ãM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352488258048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223598784199680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168140148708352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28687221 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21639174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15173738 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22606852 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22606852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221246278902272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15173742 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6465245 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51294073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22606850 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220980926494410 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218634994806208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51294073 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚191 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51283914 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28687221 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165717477053952 _step‚233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚2849 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 $ _timestamp‚1.784494750838684e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2352399590912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21638983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28677064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22606850 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223598678493184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221246278902272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6465432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220981030320308 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2422577544704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2345948727562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223598784199680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2422448108032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538œ,rÍáú èM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21638983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28677064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22606850 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223598678493184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221246278902272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6465432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220981030320308 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2422577544704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2345948727562 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223598784199680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2422448108032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352488258048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223598784199680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168140148708352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28687221 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21639174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15173738 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22606852 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22606852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221246278902272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15173742 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6465245 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51294073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22606850 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220980926494410 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218634994806208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51294073 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚191 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51283914 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28687221 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165717477053952 _step‚233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 trainer/global_step‚2849 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 $ _timestamp‚1.784494750838684e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2352399590912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812éò;Y2û - _wandbruntime‚1907  _runtime‚1907?;µü ÿ 0loss/FRt1/assignment_loss‚1.0572427131485422 2loss/SRqqt2/assignment_loss‚0.7685035493970868 .loss/FBt1/detection_loss‚0.989629828481267 /loss/FBt2/detection_loss‚0.8425960435256111 0loss/FRt2/assignment_loss‚2.4761573879587724 1loss/FBt1/assignment_loss‚0.03599433962827042 /loss/FRt1/detection_loss‚1.5090313418947447 0loss/SRqqt2/detection_loss‚1.157041769953176 trainer/global_step‚2849 _step‚234 2loss/SRqqt1/assignment_loss‚0.6600212092566964 1loss/FBt2/assignment_loss‚0.17596762262784427 &loss/total_loss‚10.906006056762935 epoch‚7 /loss/FRt2/detection_loss‚0.5718471120048976 1loss/SRqqt1/detection_loss‚0.6619731388860262 $ _timestamp‚1.784494750839684e+09 _runtime‚1907.441205698¾>i• ý „ 2loss/SRqqt1/assignment_loss‚0.6600212092566964 1loss/FBt2/assignment_loss‚0.17596762262784427 &loss/total_loss‚10.906006056762935 epoch‚7 /loss/FRt2/detection_loss‚0.5718471120048976 1loss/SRqqt1/detection_loss‚0.6619731388860262 $ _timestamp‚1.784494750839684e+09 _runtime‚1907.441205698 0loss/FRt1/assignment_loss‚1.0572427131485422 2loss/SRqqt2/assignment_loss‚0.7685035493970868 .loss/FBt1/detection_loss‚0.989629828481267 /loss/FBt2/detection_loss‚0.8425960435256111 0loss/FRt2/assignment_loss‚2.4761573879587724 1loss/FBt1/assignment_loss‚0.03599433962827042 /loss/FRt1/detection_loss‚1.5090313418947447 0loss/SRqqt2/detection_loss‚1.157041769953176 trainer/global_step‚2849 _step‚234êÀ‘MJEþ ‚(:º «õôÒ¹ÿ¾ disk.md0.in‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚6029.28955078125% disk.nvme1n1.out‚6029.28955078125 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11034.77099609375& disk.nvme0n1.out‚11034.77099609375$ memory_percent‚97.20749855041504' proc.memory.availableMB‚ 457.5234375) disk./.usagePercent‚76.35861539506811 disk.md0.out‚ 46.45703125 cpu‚25.829275908850786 proc.cpu.threads‚17% disk./.usageGB‚1172.2956428527832# disk.dm-0.out‚15508.38916015625" disk.dm-1.out‚1455.30615234375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2957.33203125* proc.memory.percent‚18.050122261047363 network.sent‚ 4.134303e+06 network.recv‚547521¾•Þÿ ‚(:“ «õôÒ‘²°¿ gpu.0.gpu‚42.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚114.764' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.254666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gk#fD€ ‚(:¹ ºõôÒÄîê¾ disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚6188.93212890625# disk.dm-0.out‚15508.95166015625" disk.dm-1.out‚1611.40771484375* proc.memory.percent‚18.050122261047363 cpu‚25.827259274073395) disk./.usagePercent‚76.35861912217828 disk.rbd0.in‚0& disk.nvme0n1.out‚11035.67138671875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚6188.93212890625 disk.md0.in‚0 disk.md0.out‚ 46.86328125& proc.memory.availableMB‚ 457.765625 network.recv‚551774 disk.nvme0c0n1.in‚0$ memory_percent‚97.20602035522461 network.sent‚ 4.185568e+06( disk.nvme0c0n1.out‚11035.67138671875 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2957.33203125 proc.cpu.threads‚17% disk./.usageGB‚1172.2957000732422 disk.rbd0.out‚0|¼ž ‚(:“ ºõôÒ¡¤™¿ gpu.0.gpu‚91.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚117.877' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.29233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬˜?2‚ - _wandbruntime‚1939  _runtime‚1939ìõ Éƒ à ' lr-AdamW/pg2‚0.00015410005733331724 trainer/global_step‚2899 % _timestamp‚1.7844947829893596e+09 _runtime‚1939.591227757 _step‚235 ' lr-AdamW/pg1‚0.00015410005733331724̴΄ È ' lr-AdamW/pg1‚0.00015410005733331724 ' lr-AdamW/pg2‚0.00015410005733331724 trainer/global_step‚2899 % _timestamp‚1.7844947829893596e+09 _runtime‚1939.591227757 _step‚235ëWYU2… - _wandbruntime‚1939  _runtime‚1939ûKPÀà'† ÚO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227319682065920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 % _timestamp‚1.7844947829904275e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22980671 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚195 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224659721100612 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15421486 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227319682065920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2391052475392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222275230550416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22980673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2391052475392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170930702686720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15421482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224659617274826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29170239 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚224928629590528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227319576366592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21994777 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2384403763770 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22980671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 _runtime‚1939.592206019 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29160096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227319576366592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6573096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚224928612551168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÆU»°‡ ßO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 _runtime‚1939.592206019 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29160096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227319576366592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6573096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚224928612551168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227319682065920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 % _timestamp‚1.7844947829904275e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22980671 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚195 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224659721100612 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15421486 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227319682065920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2391052475392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ¹ˆ5WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222275230550416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22980673 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2391052475392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170930702686720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15421482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224659617274826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29170239 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚224928629590528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227319576366592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21994777 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2384403763770 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22980671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696ìèß×2ˆ - _wandbruntime‚1940  _runtime‚1940™Šˆ ê&‰ äM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224734795120716 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22001980 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227395511748608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22988300 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222349521075808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225003762217472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168524457563648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2385274044908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52168375 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22001789 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2391838198272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2391749531136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6575438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170987435638784 _step‚237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15426542 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52158216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391749531136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52168375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚1940.006532305 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170987657088000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844947834045432e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2385187258370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15426538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227395617455104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168524363453952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222349504036448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227395617455104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52158216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391749531136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52168375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚1940.006532305 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170987657088000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844947834045432e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2385187258370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15426538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227395617455104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168524363453952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222349504036448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227395617455104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224734795120716 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22001980 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227395511748608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22988300 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222349521075808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225003762217472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168524457563648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2385274044908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52168375 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22001789 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2391838198272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2391749531136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6575438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170987435638784 _step‚237 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15426542 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231116409622016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2430402415616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29652948 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231116515321344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225989756820016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29663091 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228686095866368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29652948 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53015069 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171275206235648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225989739780656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231116409622016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6682244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2423729081004 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2430402415616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171275112125952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228413382075234 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15674282 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22356348 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segmenth=ÿàs.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _runtime‚1972.642323619 trainer/global_step‚2949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228686112905728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228413485901020 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362123 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _step‚240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228686095866368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15674286 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231116409622016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2430402415616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29652948 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231116515321344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225989756820016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29663091 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228686095866368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29652948 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53015069 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171275206235648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225989739780656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231116409622016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6682244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2423729081004 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2430402415616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171275112125952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228413382075234 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15674282 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22356348 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _runtime‚1972.642323619 trainer/global_step‚2949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228686112905728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228413485901020 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362123 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _step‚240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228686095866368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15674286 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231192345004032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2431099471360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29662768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53032518 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231192450710528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2431188138496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23369752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228761245532672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15679342 % _timestamp‚1.7844948164536836e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228761262572032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173834920644608 _step‚241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228761262572032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53032518 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2424425789178 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53042677 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2503783450112 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22363618 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29662768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2431099471360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚156 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23369750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53042677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226064030306048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6684432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171331343963648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821uê{^L š éM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23369750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53042677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226064030306048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6684432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171331343963648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231192345004032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2431099471360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29662768 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53032518 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231192450710528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2431188138496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23369752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228761245532672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15679342 % _timestamp‚1.7844948164536836e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228761262572032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173834920644608 _step‚241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228761262572032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53032518 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2424425789178 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53042677 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2503783450112 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22363618 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29662768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2431099471360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚156 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ñCŽjc2› - _wandbruntime‚1973  _runtime‚1973Øá‹œ ý &loss/total_loss‚13.920639018802426 $ _timestamp‚1.784494816454761e+09 0loss/FRt1/assignment_loss‚1.1909368592417073 2loss/SRqqt1/assignment_loss‚0.6872305230168225 .loss/FRt1/detection_loss‚1.543149095224842 /loss/FBt2/detection_loss‚3.9153773540797894 trainer/global_step‚2949 0loss/FRt2/assignment_loss‚2.4582688973138205 0loss/FBt2/assignment_loss‚0.2457550792324375 0loss/SRqqt2/detection_loss‚1.160511430134957 _runtime‚1973.056310017 /loss/FRt2/detection_loss‚0.6187306938593715 epoch‚7 _step‚242 2loss/SRqqt2/assignment_loss‚0.6947487570334668 1loss/FBt1/assignment_loss‚0.13368319473212545 1loss/SRqqt1/detection_loss‚0.6740668607462629 .loss/FBt1/detection_loss‚0.598180274186823}Ò0 ‚ 0loss/FRt2/assignment_loss‚2.4582688973138205 0loss/FBt2/assignment_loss‚0.2457550792324375 0loss/SRqqt2/detection_loss‚1.160511430134957 _runtime‚1973.056310017 /loss/FRt2/detection_loss‚0.6187306938593715 epoch‚7 _step‚242 2loss/SRqqt2/assignment_loss‚0.6947487570334668 1loss/FBt1/assignment_loss‚0.13368319473212545 1loss/SRqqt1/detection_loss‚0.6740668607462629 .loss/FBt1/detection_loss‚0.598180274186823 &loss/total_loss‚13.920639018802426 $ _timestamp‚1.784494816454761e+09 0loss/FRt1/assignment_loss‚1.1909368592417073 2loss/SRqqt1/assignment_loss‚0.6872305230168225 .loss/FRt1/detection_loss‚1.543149095224842 /loss/FBt2/detection_loss‚3.9153773540797894 trainer/global_step‚2949òàrÌAž ‚(:¶ çõôÒÀÅ„¿ disk.nvme1n1.in‚0 disk.md0.out‚ 48.12109375 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚97.20666408538818( proc.memory.availableMB‚ 457.66015625 cpu‚25.820996329896087 network.recv‚563551$ disk./.usageGB‚1169.752040863037% disk.nvme1n1.out‚6652.16259765625# disk.dm-0.out‚15965.92724609375 disk.dm-1.in‚0! disk.dm-1.out‚1990.5087890625' disk.nvme0c0n1.out‚11408.1982421875# proc.memory.rssMB‚ 2957.33203125( disk./.usagePercent‚76.1929354087632 disk.dm-0.in‚0' disk.nvme1c1n1.out‚6652.16259765625 proc.cpu.threads‚17 network.sent‚ 4.269696e+06 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚11408.1982421875* proc.memory.percent‚18.050122261047363)»¹‘Ÿ ‚(:† çõôÒ­š·¿ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.516' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/CLo;  ‚(:° öõôÒ•‹ì¾ network.recv‚567496% disk./.usageGB‚1169.7517280578613 disk.nvme1n1.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚11413.94580078125( proc.memory.availableMB‚ 457.66015625 proc.cpu.threads‚17) disk./.usagePercent‚76.19291503389424 disk.md0.in‚0 disk.dm-0.in‚0# disk.dm-0.out‚15971.23583984375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2957.33203125! disk.dm-1.out‚1990.5869140625 network.sent‚ 4.318078e+06 disk.nvme1c1n1.in‚0# disk.nvme1c1n1.out‚ 6652.8046875 disk.md0.out‚ 48.5859375 disk.nvme0n1.in‚0& disk.nvme0n1.out‚11413.94580078125! disk.nvme1n1.out‚ 6652.8046875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ memory_percent‚97.20666408538818* proc.memory.percent‚18.050122261047363 cpu‚25.81897798400466NLÓž¡ ‚(:“ öõôÒ¾«®¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚224.518' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.83933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uȇ2¢ - _wandbruntime‚2006  _runtime‚2006oê4Ç£ Á _runtime‚2006.202279657 _step‚243 & lr-AdamW/pg1‚0.0001535891847349637 & lr-AdamW/pg2‚0.0001535891847349637 trainer/global_step‚2999 % _timestamp‚1.7844948495995715e+09Õ©ËÓ̤ Æ % _timestamp‚1.7844948495995715e+09 _runtime‚2006.202279657 _step‚243 & lr-AdamW/pg1‚0.0001535891847349637 & lr-AdamW/pg2‚0.0001535891847349637 trainer/global_step‚2999óà´¢2¥ - _wandbruntime‚2006  _runtime‚2006¹Ð0žà'¦ ÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23743573 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229704283089616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53899516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23743571 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2469752355840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23743571 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232167250701428 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _step‚244 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232443579181568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22717161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6790075 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30155943 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15927082 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229704266050256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232167146875642 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23743573 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176625381200384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543382674432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7844948496004221e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232443596220928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2469663695872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234913242877440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2469752355840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2469663695872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174082092635648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2462880825386 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30145800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 trainer/global_step‚2999 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232443579181568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174081998525952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6789905 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚170 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176625596365312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30155943 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232443596220928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960i™Ä÷å'§ ßO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚170 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176625596365312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30155943 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232443596220928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23743573 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229704283089616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53899516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23743571 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2469752355840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23743571 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232167250701428 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _step‚244 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232443579181568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22717161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6790075 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30155943 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15927082 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229704266050256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232167146875642 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23743573 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176625381200384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543382674432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7844948496004221e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232443596220928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2469663695872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234913242877440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2469752355840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2469663695872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174082092635648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2462880825386 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30145800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 trainer/global_step‚2999 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232443579181568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174081998525952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6789905 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2470538078720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2470449411584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23751202 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176682555485184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2463751106524 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30155620 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53916979 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232242220895634 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229778573615008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232518728847872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234989178259456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30165777 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚6ªIKÓ82 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232518728847872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22724442 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15932142 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2463664319986 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22724257 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 % _timestamp‚1.7844948500181212e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2470449411584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2544193587712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174138136253952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174138230363648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23751200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15932138 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2544325121536 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229778556575648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6792119 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30155620 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232518745887232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232518745887232‡Öï&ª éM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6792119 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30155620 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232518745887232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232518745887232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2470538078720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2470538078720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2470449411584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23751202 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176682555485184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2463751106524 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30155620 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53916979 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232242220895634 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229778573615008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232518728847872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234989178259456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30165777 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232518728847872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22724442 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15932142 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2463664319986 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22724257 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 % _timestamp‚1.7844948500181212e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2470449411584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2544193587712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174138136253952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174138230363648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23751200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15932138 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2544325121536 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229778556575648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248õ7w©2« - _wandbruntime‚2006  _runtime‚2006ßè'¬  2loss/SRqqt2/assignment_loss‚0.8008860198909521 /loss/FBt1/detection_loss‚0.5664900196803031 trainer/global_step‚2999 0loss/FRt2/assignment_loss‚2.4799938361268294 2loss/SRqqt1/assignment_loss‚0.6795851774830318 /loss/FRt2/detection_loss‚0.5722768983936035 1loss/SRqqt1/detection_loss‚0.6828649082629129 1loss/FBt1/assignment_loss‚0.03676883730055017 1loss/FBt2/assignment_loss‚0.10456674672720186 .loss/FRt1/detection_loss‚1.442948738655254 /loss/FBt2/detection_loss‚0.8710710368633493 &loss/total_loss‚10.490600997493292 _runtime‚2006.620809035 _step‚246 1loss/SRqqt2/detection_loss‚1.2109344743087993 epoch‚7 % _timestamp‚1.7844948500192611e+09 0loss/FRt1/assignment_loss‚1.0422143038005045-;! ­ † 0loss/FRt2/assignment_loss‚2.4799938361268294 2loss/SRqqt1/assignment_loss‚0.6795851774830318 /loss/FRt2/detection_loss‚0.5722768983936035 1loss/SRqqt1/detection_loss‚0.6828649082629129 1loss/FBt1/assignment_loss‚0.03676883730055017 1loss/FBt2/assignment_loss‚0.10456674672720186 .loss/FRt1/detection_loss‚1.442948738655254 /loss/FBt2/detection_loss‚0.8710710368633493 &loss/total_loss‚10.490600997493292 _runtime‚2006.620809035 _step‚246 1loss/SRqqt2/detection_loss‚1.2109344743087993 epoch‚7 % _timestamp‚1.7844948500192611e+09 0loss/FRt1/assignment_loss‚1.0422143038005045 2loss/SRqqt2/assignment_loss‚0.8008860198909521 /loss/FBt1/detection_loss‚0.5664900196803031 trainer/global_step‚2999öVàµï<® ‚(:± …öôҫ祿% disk.nvme1n1.out‚6654.81884765625! disk.dm-0.out‚15985.349609375 cpu‚25.816972387655575 network.sent‚ 4.336773e+06( disk./.usagePercent‚76.1929458446717% disk./.usageGB‚1169.7522010803223 disk.md0.out‚ 49.02734375 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚11427.0888671875* proc.memory.percent‚18.050122261047363 network.recv‚571016 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0# proc.memory.availableMB‚457.625 proc.cpu.threads‚17% disk.nvme0n1.out‚11427.0888671875 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚1990.68896484375$ memory_percent‚97.20687866210938# proc.memory.rssMB‚ 2957.33203125' disk.nvme1c1n1.out‚6654.81884765625 disk.nvme1n1.in‚0H°'“¯ ‚(:ˆ …öôÒèÃÙ¿ gpu.0.gpu‚72.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚157.296' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚52.432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü­„±B° ‚(:· ”öôÒ¤®…¿% disk./.usageGB‚1172.5127410888672 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 49.26171875 disk.dm-0.in‚0# memory_percent‚97.2085952758789* proc.memory.percent‚18.050122261047363 network.sent‚ 4.385159e+06 network.recv‚574990) disk./.usagePercent‚76.37275629955168( disk.nvme0c0n1.out‚13341.54248046875& disk.nvme0n1.out‚13341.54248046875% disk.nvme1n1.out‚6655.28173828125" disk.dm-1.out‚1990.79443359375% proc.memory.availableMB‚ 457.34375 disk.md0.in‚0# disk.dm-0.out‚17899.64111328125# proc.memory.rssMB‚ 2957.33203125 cpu‚25.815137717956183 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚6655.28173828125 disk.dm-1.in‚0ïhžt± ‚(:„ ”öôÒ¨ôÈ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.45' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.15 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öd£2² - _wandbruntime‚2039  _runtime‚2039tà ɳ à _runtime‚2039.245526218 _step‚247 ' lr-AdamW/pg1‚0.00015332667210149175 ' lr-AdamW/pg2‚0.00015332667210149175 trainer/global_step‚3049 % _timestamp‚1.7844948826437154e+09~“»Î´ È % _timestamp‚1.7844948826437154e+09 _runtime‚2039.245526218 _step‚247 ' lr-AdamW/pg1‚0.00015332667210149175 ' lr-AdamW/pg2‚0.00015332667210149175 trainer/global_step‚3049÷ Ô¹2µ - _wandbruntime‚2039  _runtime‚2039ŽÑìà'¶ ÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24125023 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _runtime‚2039.246482799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179472953461248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2509013636096 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54763673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16179882 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179472727810560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2583842884608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚233418809359216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238710181832192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30638652 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 _step‚248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30638652 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6897421 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30648795 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233418792319856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238710076132864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238710076132864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23077307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2583974425600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236201062496768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚3049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24125023 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176888884925952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235920911676050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 % _timestamp‚1.7844948826446881e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24125021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6897249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238710181832192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236201079536128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23077131 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2509102296064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2509013636096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54773818 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236201079536128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 _step‚248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30638652 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6897421 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30648795 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233418792319856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238710076132864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238710076132864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23077307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2583974425600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236201062496768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚3049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24125023 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176888884925952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235920911676050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 % _timestamp‚1.7844948826446881e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24125021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6897249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238710181832192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236201079536128 WGDeviceStatsMonitor.on_train_batch_staÜHÐrt/allocated_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23077131 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2509102296064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2509013636096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54773818 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236201079536128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54763673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16179882 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179472727810560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2583842884608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚233418809359216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238710181832192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30638652 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30658629 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2584650783232 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24132652 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236276212163072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233493099884608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235996089521940 % _timestamp‚1.7844948830603356e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54791281 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176945022653952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235995985696042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236276229202432 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233493082845248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502989637332 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30648472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179529894886400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238786117221376 _step‚249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176945116763648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2584778122752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24132652 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24132650 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54791281 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6899429 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179529673437184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23084204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16184938 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54781122 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502902850794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2509888018944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30658629 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30648472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2509799351808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236276212163072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898]®-Çï&º éM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2509799351808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30648472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179529894886400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238786117221376 _step‚249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176945116763648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2584778122752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24132652 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24132650 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54791281 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6899429 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179529673437184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23084204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16184938 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54781122 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502902850794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2509888018944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30658629 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30648472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2509799351808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236276212163072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30658629 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2584650783232 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24132652 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236276212163072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233493099884608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235996089521940 % _timestamp‚1.7844948830603356e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54791281 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176945022653952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235995985696042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236276229202432 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233493082845248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502989637332 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24506473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242507015087616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844949157107518e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16432686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2548363576320 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242506909388288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2541444673428 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24506473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24506471 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚252 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24506473 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242507015087616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844949157107518e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16432686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2548363576320 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242506909388288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2541444673428 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24506473 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24506471 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚252 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31151481 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚165 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237207609114848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7006076 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16437738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 trainer/global_step‚3099 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2549149292032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179752003163648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23443814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31141324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚253 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24514102 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7844949161233716e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55665583 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2625111156224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24514102 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2549237959168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2549237959168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16437742 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240033695478272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2625234301440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239749750496450 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237207626154208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2542228168140 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24514100 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55665583 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242582844770304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7006241 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2072.725375718 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55655424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31141324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239749854322348 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179751909053952 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55665583 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242582844770304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7006241 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2072.725375718 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55655424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31141324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239749854322348 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179751909053952 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.a&WPqll.peak‚ 12106884175 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31151481 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚165 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237207609114848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7006076 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16437738 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 trainer/global_step‚3099 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2549149292032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179752003163648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23443814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31141324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _step‚253 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24514102 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7844949161233716e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55665583 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2625111156224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24514102 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2549237959168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2549237959168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16437742 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240033695478272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2625234301440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239749750496450 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237207626154208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2542228168140 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24514100 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136ý­#¶2Í - _wandbruntime‚2072  _runtime‚2072øðvrÎ  2loss/SRqqt1/assignment_loss‚0.7313584507670426 1loss/SRqqt2/detection_loss‚1.2038161410297714 _step‚254 /loss/FRt1/detection_loss‚1.4564829764576814 1loss/SRqqt1/detection_loss‚0.7236848915228391 _runtime‚2072.725986439 0loss/FRt1/assignment_loss‚1.0857696999771318 -loss/FRt2/detection_loss‚0.59432166552559 /loss/FBt1/detection_loss‚0.9292230624145591 epoch‚7 trainer/global_step‚3099 0loss/FRt2/assignment_loss‚2.3923922525907657 2loss/SRqqt2/assignment_loss‚0.7802568436802377 2loss/FBt1/assignment_loss‚0.021610142203719894 1loss/FBt2/assignment_loss‚0.10390249424899851 /loss/FBt2/detection_loss‚0.8987517183642282 &loss/total_loss‚10.921570338782566 % _timestamp‚1.7844949161244352e+09¼[;ã Ï † 2loss/SRqqt1/assignment_loss‚0.7313584507670426 1loss/SRqqt2/detection_loss‚1.2038161410297714 _step‚254 /loss/FRt1/detection_loss‚1.4564829764576814 1loss/SRqqt1/detection_loss‚0.7236848915228391 _runtime‚2072.725986439 0loss/FRt1/assignment_loss‚1.0857696999771318 -loss/FRt2/detection_loss‚0.59432166552559 /loss/FBt1/detection_loss‚0.9292230624145591 epoch‚7 trainer/global_step‚3099 0loss/FRt2/assignment_loss‚2.3923922525907657 2loss/SRqqt2/assignment_loss‚0.7802568436802377 2loss/FBt1/assignment_loss‚0.021610142203719894 1loss/FBt2/assignment_loss‚0.10390249424899851 /loss/FBt2/detection_loss‚0.8987517183642282 &loss/total_loss‚10.921570338782566 % _timestamp‚1.7844949161244352e+09þZBåí9Ð ‚(:® ÐöôÒôûò¾% disk./.usageGB‚1172.1840362548828 disk.nvme1n1.in‚0 disk.md0.out‚51.125 disk.dm-0.out‚18789.70703125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚14117.2177734375( disk./.usagePercent‚76.3513457909059% disk.nvme1n1.out‚6774.10302734375" disk.dm-1.out‚1991.24853515625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚14117.2177734375' disk.nvme1c1n1.out‚6774.10302734375# memory_percent‚97.2053050994873 disk.dm-0.in‚0 cpu‚25.807681213777588 network.recv‚588901 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 457.8828125# proc.memory.rssMB‚ 2957.33203125* proc.memory.percent‚18.050122261047363 proc.cpu.threads‚17 network.sent‚ 4.462267e+06â<Ç”ŸÑ ‚(:” ÐöôÒ­ûÒ¿ gpu.0.gpu‚90.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚161.053' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚53.684333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôªJUDÒ ‚(:¹ ßöôÒÊ…Š¿& disk.nvme0n1.out‚14122.38916015625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚6776.15185546875# proc.memory.rssMB‚ 2957.33203125* proc.memory.percent‚18.050122261047363 cpu‚25.806084638956502 network.sent‚ 4.524573e+06) disk./.usagePercent‚76.35136641424887# disk.dm-0.out‚18796.08056640625 disk.dm-1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 458.1328125 network.recv‚593636 disk.rbd0.in‚0 disk.nvme0n1.in‚0$ memory_percent‚97.20377922058105% disk./.usageGB‚1172.1843528747559 disk.md0.in‚0 disk.md0.out‚ 51.4453125 disk.dm-0.in‚0" disk.dm-1.out‚1991.39794921875 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚6776.15185546875 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14122.38916015625HêÚTžÓ ‚(:“ ßöôÒ‰Üο gpu.0.gpu‚72.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.262' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.08733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…mO2Ô - _wandbruntime‚2105  _runtime‚2105yœ]ÉÕ Ã _step‚255 ' lr-AdamW/pg1‚0.00015278757846835407 ' lr-AdamW/pg2‚0.00015278757846835407 trainer/global_step‚3149 % _timestamp‚1.7844949486726058e+09 _runtime‚2105.274467948È7ÖïÎÖ È ' lr-AdamW/pg1‚0.00015278757846835407 ' lr-AdamW/pg2‚0.00015278757846835407 trainer/global_step‚3149 % _timestamp‚1.7844949486726058e+09 _runtime‚2105.274467948 _step‚255ÿñçô2× - _wandbruntime‚2105  _runtime‚2105W¼'‘à'Ø ÚO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56512277 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229844992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56522422 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31634499 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7110625 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2664828585984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24887921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31624356 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243716046166528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2587713516544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24887923 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2664964321280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2580683204236 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16685482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 _step‚256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23796286 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182502751835648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135735296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240847861898416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2587802176512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2105.279308166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2587713516544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7110800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246303848343040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2580596417810 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16685486 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246303848343040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246303742643712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185167486311936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240847844859056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31624356 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23796107 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31634499 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246303742643712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243428545102652 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24887921 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243716029127168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24887923 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246303848343040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246303742643712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185167486311936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240847844859056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31624356 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23796107 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31634499 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246303742643712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243428545102652 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24887921 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243716029127168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24887923 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24895550 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243791178793472 UHDeviceStatsMoå6jR%nitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56539885 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚169 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240922152423808 _step‚257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240922135384448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229837824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚3149 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588587899392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23803346 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2665636091392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56539885 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31644333 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243791195832832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246379678025728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7112635 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243503619122756 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243791195832832 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚2105.689359332 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24895550 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23803173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185224661383168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24895552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844949490873508e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246379783732224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185224431545344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24895552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56529726 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56529726 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243503515296858 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16690542 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2581379912410 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246379783732224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182558795453952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2581466698948 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588587899392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2588499232256 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2581466698948 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588587899392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2588499232256 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24895550 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243791178793472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56539885 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚169 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240922152423808 _step‚257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240922135384448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229837824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚3149 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588587899392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23803346 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2665636091392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56539885 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31644333 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚243791195832832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246379678025728 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7112635 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243503619122756 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243791195832832 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚2105.689359332 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24895550 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23803173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185224661383168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24895552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚173 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 147779584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844949490873508e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246379783732224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185224431545344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24895552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56529726ˆÒÒa2Ý - _wandbruntime‚2105  _runtime‚2105`Óž½Þ € % _timestamp‚1.7844949490884714e+09 _runtime‚2105.690227523 /loss/FBt2/detection_loss‚0.7161613608257046 epoch‚7 1loss/SRqqt2/detection_loss‚1.0823700397169043 trainer/global_step‚3149 0loss/FRt1/assignment_loss‚1.0441085254619766 /loss/FBt1/detection_loss‚0.9491377432245465 _step‚258 %loss/total_loss‚10.57685823936816 0loss/FRt2/assignment_loss‚2.3587509796376334 2loss/SRqqt1/assignment_loss‚0.7031154080051192 2loss/SRqqt2/assignment_loss‚0.7292632855657024 0loss/FBt1/assignment_loss‚0.0433452561047917 1loss/FBt2/assignment_loss‚0.24590555752347734 /loss/FRt2/detection_loss‚0.5961698258689531 1loss/SRqqt1/detection_loss‚0.6963173533952087 /loss/FRt1/detection_loss‚1.41221290403814374ä9 ß … /loss/FBt2/detection_loss‚0.7161613608257046 epoch‚7 1loss/SRqqt2/detection_loss‚1.0823700397169043 trainer/global_step‚3149 0loss/FRt1/assignment_loss‚1.0441085254619766 /loss/FBt1/detection_loss‚0.9491377432245465 _step‚258 %loss/total_loss‚10.57685823936816 0loss/FRt2/assignment_loss‚2.3587509796376334 2loss/SRqqt1/assignment_loss‚0.7031154080051192 2loss/SRqqt2/assignment_loss‚0.7292632855657024 0loss/FBt1/assignment_loss‚0.0433452561047917 1loss/FBt2/assignment_loss‚0.24590555752347734 /loss/FRt2/detection_loss‚0.5961698258689531 1loss/SRqqt1/detection_loss‚0.6963173533952087 /loss/FRt1/detection_loss‚1.4122129040381437 % _timestamp‚1.7844949490884714e+09 _runtime‚2105.690227523‚^>œ%Dà ‚(:¹ îöôÒÔùª¿& disk.nvme0n1.out‚14155.43017578125 disk.nvme1c1n1.in‚0 disk.md0.in‚0# proc.memory.rssMB‚ 2957.33203125* proc.memory.percent‚18.050122261047363 network.sent‚ 4.542241e+06 network.recv‚596866 disk.rbd0.in‚0 disk.nvme1n1.in‚0$ memory_percent‚97.20408916473389( proc.memory.availableMB‚ 458.08203125 cpu‚25.80427374199243$ disk./.usageGB‚1172.486930847168 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚6776.66943359375 disk.dm-0.in‚0# disk.dm-0.out‚18828.88134765625 disk.dm-1.in‚0" disk.dm-1.out‚1991.47998046875 proc.cpu.threads‚17) disk./.usagePercent‚76.37107512438807 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk.nvme1n1.out‚6776.66943359375 disk.md0.out‚ 51.74609375( disk.nvme0c0n1.out‚14155.43017578125Øxdžá ‚(:“ îöôÒÜåÝ¿ gpu.0.gpu‚98.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚116.249' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.74966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÌÓÝEâ ‚(:º ýöôÒÜÔç¾ disk.rbd0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚97.20418453216553( proc.memory.availableMB‚ 458.06640625 network.sent‚ 4.592489e+06% disk./.usageGB‚1172.4867477416992% disk.nvme1n1.out‚6777.38818359375 disk.md0.out‚ 52.13671875 disk.nvme0n1.in‚0# proc.memory.rssMB‚ 2957.33203125* proc.memory.percent‚18.050122261047363 network.recv‚601443 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17' disk.nvme1c1n1.out‚6777.38818359375 disk.nvme1n1.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚14182.68310546875& disk.nvme0n1.out‚14182.68310546875 cpu‚25.80250789530304) disk./.usagePercent‚76.37106319763551 disk.md0.in‚0# disk.dm-0.out‚18855.86474609375" disk.dm-1.out‚1991.63232421875¸‰$Øã ‚(:’ ýöôÒ½£•¿ gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.772' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.257333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sY¥2ä - _wandbruntime‚2138  _runtime‚2138ZbÅå ¿ % _timestamp‚1.7844949817640572e+09 _runtime‚2138.365928383 _step‚259 % lr-AdamW/pg1‚0.000152511032093665 % lr-AdamW/pg2‚0.000152511032093665 trainer/global_step‚3199ô;T)Êæ Ä % lr-AdamW/pg1‚0.000152511032093665 % lr-AdamW/pg2‚0.000152511032093665 trainer/global_step‚3199 % _timestamp‚1.7844949817640572e+09 _runtime‚2138.365928383 _step‚259ƒ.–ï2ç - _wandbruntime‚2138  _runtime‚21384Øfòc!è ÚO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚2138.367036115 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25269373 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32127351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250100575899136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250100575899136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2705426349568 % _timestamp‚1.7844949817651434e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185309544125952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2619834948618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247473512442368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250100681598464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188015064585216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188014841031680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2627152116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚3199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25269371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32127351 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32117208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2445623881680169kXå'é ßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32127351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 147779584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250100575899136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250100575899136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2705426349568 % _timestamp‚1.7844949817651434e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185309544125952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2619834948618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247473512442368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250100681598464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188015064585216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188014841031680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2627152116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚3199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25269371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32127351 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32117208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚244562388168016 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚2138.367036115 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25269373 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840„…ÚÚ‘2ê - _wandbruntime‚2138  _runtime‚2138Ëš}ê&ë äM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24162582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16943342 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25277000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250176616987648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32137185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2627937839616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247548662108672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32127028 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32137185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2627849172480 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627849172480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244636678693408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7219240 trainer/global_step‚3199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185365775963648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250176616987648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57414187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188071783356928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2627937839616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚2138.786684615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247257383923164 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚261 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16943338 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25277002 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2620618443218 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247548679148032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250176511281152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57414187 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57404028 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185365681853952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247548679148032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25277002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7844949821846538e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2620705229756 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2706101502976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25277000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24162403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247548662108672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374œEükï&ì éM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247257280097266 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244636661654048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57414187 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57404028 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185365681853952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚247548679148032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25277002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7844949821846538e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2620705229756 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2706101502976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25277000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24162403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247548662108672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24162582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16943342 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25277000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250176616987648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32137185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2627937839616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247548662108672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32127028 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32137185 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2627849172480 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627849172480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244636678693408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7219240 trainer/global_step‚3199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185365775963648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250176616987648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57414187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188071783356928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2627937839616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚2138.786684615 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247257383923164 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚261 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16943338 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25277002 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2620618443218 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247548679148032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250176511281152 Fñ ‚(:… Œ÷ôÒʑ¿ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.18' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.06 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†€%¿2ò - _wandbruntime‚2152  _runtime‚2152‰7qȼ=ó ¶{ 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8259998559951782 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9854498505592346 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_1_of_1‚0.34059759974479675 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9607586860656738 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09699348360300064 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31228259205818176 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8259636759757996 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7427166700363159 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54708331823349 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05740193650126457 *jet/accuracy_2_of_4‚0.7974003553390503 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6599705219268799 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _step‚263 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21828503906726837 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.790727436542511 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15820631384849548 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09785491973161697 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949482679367065 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16894105076789856 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19311362504959106 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 /particle/accuracy_4_of_4‚0.9554592967033386 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9075402617454529 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5643800497055054 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.380754292011261 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2300628274679184 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚9.993193218219538 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900612831115723 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1421188861131668 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6173879504203796 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967741370201111 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9840649366378784 ,jet/accuracy_4_of_4‚0.011098379269242287 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6557715535163879 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.790131151676178 particle/accuracy_1_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416269302368 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6344676613807678 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5549708604812622 *jet/accuracy_1_of_4‚0.9671741724014282 +particle/sensitivity‚0.6301157474517822 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7273207902908325 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4759864807128906 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6264399886131287 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22900603711605072 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3793637752532959 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2167433351278305 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873273968696594 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7385064363479614 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8195200562477112 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5649042725563049 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22079293429851532 *jet/accuracy_1_of_2‚0.7965167164802551 particle/accuracy_1_of_1‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0479063019156456 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7142829298973083 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970446825027466 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8966090679168701 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6335121393203735 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992041826248169 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27232274413108826 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03013649769127369 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6378609538078308 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03685522451996803 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5858989953994751 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14379432797431946 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 2152.95158417 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24643079936504364 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6732115149497986 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20453236997127533 particle/accuracy_5_of_5‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24891254305839539 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22900690138339996 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5693478584289551 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7093139886856079 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9747205376625061 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6853657960891724 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12801499664783478 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8479701280593872 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6400076746940613 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3878760039806366 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6867050528526306 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3809491693973541 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2279493659734726 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17212116718292236 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14058296382427216 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8227890729904175 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6402963399887085 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09145238995552063 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9331170320510864 +jet/accuracy_3_of_4‚0.09845653921365738 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 $ _timestamp‚1.784494996349312e+09 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07898011803627014 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6072582602500916 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2026345580816269 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2549217939376831 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8429608345031738 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6442670226097107 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24405725300312042 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33181652426719666 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967741370201111 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03788239508867264 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.779670000076294 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6034836769104004 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39050111174583435 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957668781280518 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6415191888809204 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8783701062202454 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48593536019325256 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2655230164527893 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07898011803627014 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9906336665153503 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2662920355796814 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6804804801940918 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3333333134651184 epoch‚7 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09089924395084381 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2528357207775116 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9878179430961609 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7462940216064453 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870947003364563 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6442670226097107 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8229780793190002 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.696277379989624 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8080748319625854 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47657716274261475 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7779920697212219 particle/accuracy_3_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7639078497886658 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07089954614639282 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3769865036010742 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7348538637161255 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6784856915473938 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6825491786003113 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29785221815109253 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6893122792243958 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6124058961868286 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6476013660430908 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2648749351501465 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9690577983856201 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 trainer/global_step‚3199 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2549217939376831 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8080748319625854 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6217169761657715 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8833882212638855 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.966558575630188 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4658510982990265 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.9237214922904968 +jet/accuracy_3_of_3‚0.06805956363677979ÞæuÍ+:ô »{ ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 /particle/accuracy_4_of_4‚0.9554592967033386 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9075402617454529 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5643800497055054 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.380754292011261 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2300628274679184 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚9.993193218219538 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9900612831115723 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1421188861131668 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6173879504203796 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967741370201111 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9840649366378784 ,jet/accuracy_4_of_4‚0.011098379269242287 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6557715535163879 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.790131151676178 particle/accuracy_1_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416269302368 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6344676613807678 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5549708604812622 *jet/accuracy_1_of_4‚0.9671741724014282 +particle/sensitivity‚0.6301157474517822 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7273207902908325 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4759864807128906 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6264399886131287 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22900603711605072 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3793637752532959 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2167433351278305 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873273968696594 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7385064363479614 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8195200562477112 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5649042725563049 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22079293429851532 *jet/accuracy_1_of_2‚0.7965167164802551 particle/accuracy_1_of_1‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0479063019156456 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7142829298973083 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970446825027466 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8966090679168701 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6335121393203735 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992041826248169 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27232274413108826 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03013649769127369 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6378609538078308 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03685522451996803 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5858989953994751 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14379432797431946 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 2152.95158417 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24643079936504364 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6732115149497986 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20453236997127533 particle/accuracy_5_of_5‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24891254305839539 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22900690138339996 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5693478584289551 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7093139886856079 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9747205376625061 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6853657960891724 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12801499664783478 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8479701280593872 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6400076746940613 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3878760039806366 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6867050528526306 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3809491693973541 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2279493659734726 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17212116718292236 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14058296382427216 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8227890729904175 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6402963399887085 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09145238995552063 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9331170320510864 +jet/accuracy_3_of_4‚0.09845653921365738 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 $ _timestamp‚1.784494996349312e+09 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07898011803627014 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6072582602500916 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2026345580816269 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2549217939376831 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8429608345031738 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6442670226097107 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24405725300312042 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33181652426719666 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967741370201111 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03788239508867264 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.779670000076294 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6034836769104004 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39050111174583435 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957668781280518 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6415191888809204 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8783701062202454 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48593536019325256 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2655230164527893 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07898011803627014 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9906336665153503 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2662920355796814 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6804804801940918 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3333333134651184 epoch‚7 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09089924395084381 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2528357207775116 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9878179430961609 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7462940216064453 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870947003364563 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6442670226097107 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8229780793190002 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.696277379989624 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8080748319625854 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518517911434174 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47657716274261475 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7779920697212219 particle/accuracy_3_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7639078497886658 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07089954614639282 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3769865036010742 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7348538637161255 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6784856915473938 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6825491786003113 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29785221815109253 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6893122792243958 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518517911434174 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6124058961868286 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6476013660430908 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2648749351501465 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9690577983856201 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 trainer/global_step‚3199 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2549217939376831 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8080748319625854 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6217169761657715 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8833882212638855 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.966558575630188 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4658510982990265 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.9237214922904968 +jet/accuracy_3_of_3‚0.06805956363677979 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8259998559951782 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9854498505592346 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_1_of_1‚0.34059759974479675 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9607586860656738 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09699348360300064 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31228259205818176 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8259636759757996 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7427;·vv–166700363159 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.54708331823349 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05740193650126457 *jet/accuracy_2_of_4‚0.7974003553390503 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6599705219268799 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _step‚263 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21828503906726837 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.790727436542511 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15820631384849548 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09785491973161697 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949482679367065 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16894105076789856 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19311362504959106‡ÈĬì=õ ‚(:² ›÷ôÒµ‹†¿% disk./.usageGB‚1169.7051734924316 disk.md0.out‚ 52.9609375 disk.rbd0.out‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚14358.46142578125 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚6875.654296875) disk./.usagePercent‚76.18988265705465 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚14358.46142578125$ memory_percent‚97.21639156341553( proc.memory.availableMB‚ 456.06640625# proc.memory.rssMB‚ 2959.07421875 network.sent‚ 4.657773e+06 disk.md0.in‚0 disk.dm-0.out‚19064.03515625" disk.dm-1.out‚2055.72802734375% disk.nvme1c1n1.out‚6875.654296875 disk.nvme1n1.in‚0* proc.memory.percent‚18.060755729675293 disk.dm-0.in‚0 cpu‚25.81822257808025 proc.cpu.threads‚17 network.recv‚609211ŒÍþü“ö ‚(:ˆ ›÷ôÒŒ˜å¿ gpu.0.gpu‚60.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚207.624' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚69.208 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0줴B:÷ ‚(:¯ ª÷ôÒÃó–¿ network.recv‚613076 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0( proc.memory.availableMB‚ 327.30078125 proc.cpu.threads‚17 network.sent‚ 4.716471e+06% disk./.usageGB‚1169.7366981506348% disk.nvme1n1.out‚6924.56396484375 disk.md0.in‚0 disk.dm-0.in‚0 cpu‚25.81626697789901 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 14364.3828125' disk.nvme1c1n1.out‚6924.56396484375 disk.md0.out‚ 53.33984375 disk.dm-0.out‚19069.92578125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2959.07421875* proc.memory.percent‚18.060755729675293) disk./.usagePercent‚76.19193604628789 disk.nvme1n1.in‚0" disk.dm-1.out‚2103.84130859375 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 14364.3828125$ memory_percent‚98.00231456756592mµHnžø ‚(:“ ª÷ôÒ¢æÅ¿ gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚228.059' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.01966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²E¤2ù - _wandbruntime‚2186  _runtime‚2186`‰K5Çú Á _runtime‚2186.471410586 _step‚264 & lr-AdamW/pg1‚0.0001522298429298588 & lr-AdamW/pg2‚0.0001522298429298588 trainer/global_step‚3249 % _timestamp‚1.7844950298696277e+09ꎪÌû Æ _step‚264 & lr-AdamW/pg1‚0.0001522298429298588 & lr-AdamW/pg2‚0.0001522298429298588 trainer/global_step‚3249 % _timestamp‚1.7844950298696277e+09 _runtime‚2186.471410586ˆ.Üß2ü - _wandbruntime‚2186  _runtime‚2186èXÍà'ý ÚO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2761505750016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25772621 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24671881 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2681842774016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254909325827584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17299337 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32702710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188919889840640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32692567 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252227500092928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58475333 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2761624708096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252227483053568 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118958080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252227483053568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25772621 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7372544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251924853172440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2681842774016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32692567 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252227500092928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2681931433984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254909325827584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254909431526912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58465188 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 % _timestamp‚1.7844950298703892e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚254909431526912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25772623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24672064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254909325827584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17299337 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32702710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188919889840640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32692567 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252227500092928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58475333 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2761624708096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252227483053568 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118958080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252227483053568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25772621 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7372544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251924853172440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2681842774016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32692567 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252227500092928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2681931433984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254909325827584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254909431526912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58465188 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 % _timestamp‚1.7844950298703892e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚254909431526912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25772623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24672064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚254985366916096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24679307 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682628489728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58492796 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780252 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252000031018330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675359019130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚254985366916096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682717156864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2682628489728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24679124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58492796 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249324654959840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚183 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32702387 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2675272232592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32712544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚3249 % _timestamp‚1.7844950302882943e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252302632719872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17304397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2762429715968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252302649759232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188975933458944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25780250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.0‚lqfreed‚251999927192432 _runtime‚2186.890283695 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32702387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17304393 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚254985261209600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58482637 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191738252612608 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252302632719872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2682717156864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7374910 _step‚266 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚179غÐï& éM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252302649759232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚254985366916096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24679307 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682628489728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58492796 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780252 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252000031018330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675359019130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚254985366916096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682717156864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2682628489728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24679124 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58492796 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249324654959840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚183 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32702387 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2675272232592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32712544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚3249 % _timestamp‚1.7844950302882943e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252302632719872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17304397 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2762429715968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252302649759232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188975933458944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25780250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251999927192432 _runtime‚2186.890283695 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32702387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17304393 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚254985261209600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58482637 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191738252612608 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252302632719872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2682717156864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7374910 _step‚266 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚179 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110562304Šõª¶«2‚ - _wandbruntime‚2186  _runtime‚2186l”ƒ ÿ 2loss/SRqqt1/assignment_loss‚0.6562736207477287 1loss/FBt1/assignment_loss‚0.02044864523247532 /loss/FBt2/detection_loss‚0.8720430086343965 0loss/FRt1/assignment_loss‚1.0960773683458793 0loss/FBt2/assignment_loss‚0.1598205090604041 /loss/FRt1/detection_loss‚1.4936020963568462 1loss/SRqqt1/detection_loss‚0.7514608383980004 /loss/FBt1/detection_loss‚0.6678751666390857 &loss/total_loss‚10.445090304410575 epoch‚8 trainer/global_step‚3249 /loss/FRt2/assignment_loss‚2.344080690329176 % _timestamp‚1.7844950302893775e+09 _runtime‚2186.890923996 1loss/SRqqt2/assignment_loss‚0.719476052873367 /loss/FRt2/detection_loss‚0.5647041077622116 1loss/SRqqt2/detection_loss‚1.0992282000310043 _step‚267Rùô „ „ 2loss/SRqqt1/assignment_loss‚0.6562736207477287 1loss/FBt1/assignment_loss‚0.02044864523247532 /loss/FBt2/detection_loss‚0.8720430086343965 0loss/FRt1/assignment_loss‚1.0960773683458793 0loss/FBt2/assignment_loss‚0.1598205090604041 /loss/FRt1/detection_loss‚1.4936020963568462 1loss/SRqqt1/detection_loss‚0.7514608383980004 /loss/FBt1/detection_loss‚0.6678751666390857 &loss/total_loss‚10.445090304410575 epoch‚8 trainer/global_step‚3249 /loss/FRt2/assignment_loss‚2.344080690329176 % _timestamp‚1.7844950302893775e+09 _runtime‚2186.890923996 1loss/SRqqt2/assignment_loss‚0.719476052873367 /loss/FRt2/detection_loss‚0.5647041077622116 1loss/SRqqt2/detection_loss‚1.0992282000310043 _step‚267‹VÞj=… ‚(:² ¹÷ôÒÞÕÙ¾ disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚6941.193359375 disk.md0.in‚0 disk.md0.out‚ 53.74609375 disk.dm-0.in‚0! disk.dm-0.out‚19072.783203125$ memory_percent‚98.00255298614502( proc.memory.availableMB‚ 327.26171875# proc.memory.rssMB‚ 2959.07421875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0% disk.nvme1c1n1.out‚6941.193359375* proc.memory.percent‚18.060755729675293 proc.cpu.threads‚17 network.sent‚ 4.719494e+06 network.recv‚616798% disk./.usageGB‚1169.7367057800293' disk.nvme0c0n1.out‚14367.3994140625 cpu‚25.814466522230312) disk./.usagePercent‚76.19193654323591! disk.dm-1.out‚2119.7646484375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚14367.3994140625ž†Ÿ† ‚(:’ ¹÷ôÒÏÚº¿ gpu.0.gpu‚6.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.352' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.117333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ…US9‡ ‚(:® È÷ôÒµ¼—¿! proc.memory.rssMB‚ 2959.078125 cpu‚25.812779586757742 proc.cpu.threads‚17 network.recv‚620963% disk./.usageGB‚1170.0212593078613 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 54.1015625 disk.rbd0.in‚0$ memory_percent‚98.00124168395996* proc.memory.percent‚18.060779571533203) disk./.usagePercent‚76.21047121366952 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚6941.908203125 disk.dm-0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚14368.2587890625' proc.memory.availableMB‚ 327.4765625 network.sent‚ 4.783619e+06% disk.nvme1c1n1.out‚6941.908203125! disk.dm-0.out‚19073.345703125 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚2120.0107421875' disk.nvme0c0n1.out‚14368.2587890625…dŸžˆ ‚(:“ È÷ôÒ׵ſ gpu.0.gpu‚87.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚179.682' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚59.89399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ÿ<ü2‰ - _wandbruntime‚2219  _runtime‚2219'iª°ÈŠ  trainer/global_step‚3299 $ _timestamp‚1.784495062835645e+09 _runtime‚2219.438517723 _step‚268 ' lr-AdamW/pg1‚0.00015194402903719002 ' lr-AdamW/pg2‚0.000151944029037190021N"jÍ‹ Ç _runtime‚2219.438517723 _step‚268 ' lr-AdamW/pg1‚0.00015194402903719002 ' lr-AdamW/pg2‚0.00015194402903719002 trainer/global_step‚3299 $ _timestamp‚1.784495062835645e+09Œ f—#2Œ - _wandbruntime‚2219  _runtime‚2219ö$1à' ÚO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194528839213056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33195562 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33185419 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59349635 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚191 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7481883 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2721281374208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17552137 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154071 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844950628391602e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 trainer/global_step‚3299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194529050183680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255678721798634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255678617972848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17552141 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2721192714240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 210970624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2713727268800 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258706264782336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255984983408128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚187 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191726776240640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258706159083008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2721281374208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802273943040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255984983408128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25034211 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713814055226 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33195562 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _runtime‚2219.440931827 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116860928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26154073 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252964907743408 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255984966368768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258706159083008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2721192714240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25034020 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191726682130944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258706264782336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258706159083008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2721281374208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802273943040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255984983408128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25034211 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713814055226 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33195562 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _runtime‚2219.440931827 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116860928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26154073 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252964907743408 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255984966368768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258706159083008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2721192714240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25034020 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191726682130944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258706264782336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2721281374208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17552137 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154071 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844950628391602e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 trainer/global_step‚3299i2 - _wandbruntime‚2219  _runtime‚2219]€Ó?ê& äM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26161702 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33195239 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2802965373952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59356939 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256060116035072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59367098 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258782200171520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7484181 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256060116035072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803084324864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17557197 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194585785232896 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253039198268800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25041378 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25041199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191782913968640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7484006 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33205396 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2722067097088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26161700 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255753691992840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256060133074432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26161702 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255753795818738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2722067097088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2721978429952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714597549938 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59367098 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59356939 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26161700 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191782819858944ô]¼ï&‘ éM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17557197 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194585785232896 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253039198268800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25041378 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25041199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191782913968640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7484006 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33205396 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2722067097088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26161700 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255753691992840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256060133074432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26161702 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255753795818738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2722067097088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2721978429952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714597549938 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59367098 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59356939 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26161700 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191782819858944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26161702 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33195239 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2802965373952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59356939 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256060116035072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59367098 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258782200171520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7484181 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256060116035072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803084324864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682Ž€9ý2’ - _wandbruntime‚2219  _runtime‚2219Žeö=“  epoch‚8 trainer/global_step‚3299 0loss/FRt1/assignment_loss‚0.9868792726202049 2loss/SRqqt1/assignment_loss‚0.6742128954253424 1loss/SRqqt1/detection_loss‚0.6617620735537043 1loss/SRqqt2/detection_loss‚1.1042651249788726 /loss/FBt2/detection_loss‚0.6501501750924786 % _timestamp‚1.7844950632425175e+09 /loss/FRt2/assignment_loss‚2.282412210033224 2loss/FBt1/assignment_loss‚0.001498666019562493 1loss/FBt2/assignment_loss‚0.08425642575940938 /loss/FRt2/detection_loss‚0.5758933702629574 /loss/FBt1/detection_loss‚0.7834606422312458 %loss/total_loss‚9.980932730452974 _step‚271 2loss/SRqqt2/assignment_loss‚0.7117672984016687 /loss/FRt1/detection_loss‚1.4643745760743032 _runtime‚2219.844088346ÎÌíà ” † trainer/global_step‚3299 0loss/FRt1/assignment_loss‚0.9868792726202049 2loss/SRqqt1/assignment_loss‚0.6742128954253424 1loss/SRqqt1/detection_loss‚0.6617620735537043 1loss/SRqqt2/detection_loss‚1.1042651249788726 /loss/FBt2/detection_loss‚0.6501501750924786 % _timestamp‚1.7844950632425175e+09 /loss/FRt2/assignment_loss‚2.282412210033224 2loss/FBt1/assignment_loss‚0.001498666019562493 1loss/FBt2/assignment_loss‚0.08425642575940938 /loss/FRt2/detection_loss‚0.5758933702629574 /loss/FBt1/detection_loss‚0.7834606422312458 %loss/total_loss‚9.980932730452974 _step‚271 2loss/SRqqt2/assignment_loss‚0.7117672984016687 /loss/FRt1/detection_loss‚1.4643745760743032 _runtime‚2219.844088346 epoch‚8°ŸÆ‡Ÿ• ‚(:” ×÷ô҆ʚ¿ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚118.221' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.407000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤,™:– ‚(:¯ ×÷ôҋ⪿ network.sent‚ 4.799421e+06% disk./.usageGB‚1172.2485160827637& disk.nvme0c0n1.out‚15330.408203125 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚15330.408203125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7904.35986328125 disk.dm-1.in‚0) disk./.usagePercent‚76.35554574712677 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7904.35986328125 disk.md0.in‚0 disk.md0.out‚ 54.48828125# disk.dm-0.out‚20996.29443359375 disk.dm-1.out‚ 2120.83203125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.00443649291992& proc.memory.availableMB‚ 326.953125! proc.memory.rssMB‚ 2959.078125* proc.memory.percent‚18.060779571533203 network.recv‚624340 disk.dm-0.in‚0 cpu‚25.81526405966445 proc.cpu.threads‚17ÚHÝ<— ‚(:± æ÷ôÒèà¾) disk./.usagePercent‚76.42118711460687( disk.nvme0c0n1.out‚15730.35205078125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚7953.2880859375 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚21443.68115234375$ memory_percent‚98.00152778625488! proc.memory.rssMB‚ 2959.078125 network.sent‚ 4.849095e+06& disk.nvme0n1.out‚15730.35205078125$ disk.nvme1n1.out‚7953.2880859375 disk.md0.in‚0 disk.md0.out‚ 55.09765625 disk.dm-1.in‚0 disk.dm-1.out‚ 2121.0234375* proc.memory.percent‚18.060779571533203 network.recv‚628267$ disk./.usageGB‚1173.256275177002 disk.dm-0.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 327.4296875 cpu‚25.814509351421673£‚ÙvŸ˜ ‚(:” æ÷ôÒŸü¡¿ gpu.0.gpu‚85.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.825' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.275000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uI³;™ ‚(:° õ÷ôÒèåܾ cpu‚25.813309482947968 network.sent‚ 4.854912e+06 disk.nvme0n1.in‚0& disk.nvme0n1.out‚15744.35400390625% disk.nvme1c1n1.out‚9046.548828125 disk.nvme1n1.in‚0$ memory_percent‚98.00307750701904! proc.memory.rssMB‚ 2959.078125* proc.memory.percent‚18.060779571533203 network.recv‚631616 disk.dm-0.out‚ 22549.8046875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 55.5703125( proc.memory.availableMB‚ 327.17578125% disk./.usageGB‚1174.0136451721191 disk.dm-0.in‚0" disk.dm-1.out‚2121.16357421875 disk.rbd0.out‚0# disk.nvme1n1.out‚9046.548828125 proc.cpu.threads‚17) disk./.usagePercent‚76.47051914490272 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15744.35400390625­ä·Ìžš ‚(:“ õ÷ôÒÙ·ž¿ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚221.789' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.92966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bï·V2› - _wandbruntime‚2250  _runtime‚2250ãr9ÆÉœ à _step‚272 ' lr-AdamW/pg1‚0.00015165360877295088 ' lr-AdamW/pg2‚0.00015165360877295088 trainer/global_step‚3349 % _timestamp‚1.7844950940565188e+09 _runtime‚2250.658380479ô+æÛÎ È ' lr-AdamW/pg1‚0.00015165360877295088 ' lr-AdamW/pg2‚0.00015165360877295088 trainer/global_step‚3349 % _timestamp‚1.7844950940565188e+09 _runtime‚2250.658380479 _step‚272þv‰2ž - _wandbruntime‚2250  _runtime‚2250kÒH¨1 Ÿ ÙO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262502992338432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60213792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262502992338432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26535521 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7590392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚[ää®25395152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256679416973648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33688414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2753052586034 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262503098037760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262503098037760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2760631314432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256679434013008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17804937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2752965799608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60223937 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26535523 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194533568530944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259432382773256 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259742466723328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2760542654464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259742449683968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33678271 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194533662640640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2842731356160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33688414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259742449683968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840ñ<Í+ä'  ÞO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17804937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2752965799608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60223937 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26535523 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194533568530944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259432382773256 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259742466723328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2760542654464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259742449683968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33678271 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194533662640640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2842731356160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33688414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259742449683968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262502992338432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60213792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262502992338432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26535521 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7590392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25395152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256679416973648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33688414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2753052586034 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262503098037760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262503098037760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2760631314432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256679434013008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760‘ ³@R2¡ - _wandbruntime‚2251  _runtime‚2251RÌ|¬ê&¢ äM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26543152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2761417037312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2761328370176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33698248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197433469060096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197433243416576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2753836080746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26543152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33688091 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33698248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25402513 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259817616389632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17809993 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262578927720448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚274 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256753724538400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194589800368640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60241400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844950944555936e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25402320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2843668691456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 _runtime‚2251.057560581 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26543150 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60241400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33688091 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60231241 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚189 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2843537157632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496gø^/£ éM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2843668691456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 _runtime‚2251.057560581 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26543150 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60241400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33688091 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60231241 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚189 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2843537157632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26543152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2761417037312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2761328370176 ZMDeviceStatsMo+ÉÀnitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33698248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197433469060096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197433243416576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2753836080746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26543152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33688091 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33698248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25402513 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259817616389632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17809993 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262578927720448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _step‚274 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256753724538400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194589800368640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60241400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844950944555936e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25402320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279’‘*ë2¤ - _wandbruntime‚2251  _runtime‚2251þCHÞ¥  &loss/total_loss‚10.318547403860794 epoch‚8 0loss/FRt1/assignment_loss‚1.1363202394614038 2loss/SRqqt1/assignment_loss‚0.6806658635881052 1loss/SRqqt2/detection_loss‚1.1435652218307935 2loss/SRqqt2/assignment_loss‚0.6831156445777213 /loss/FRt1/detection_loss‚1.5408280956248754 /loss/FRt2/detection_loss‚0.5634197435934789 .loss/FBt2/detection_loss‚0.592439427614158 /loss/FRt2/assignment_loss‚2.336273281736456 2loss/FBt1/assignment_loss‚0.012999611231874687 1loss/FBt2/assignment_loss‚0.10006166384931507 1loss/SRqqt1/detection_loss‚0.6799379230915791 /loss/FBt1/detection_loss‚0.8489206876610315 trainer/global_step‚3349 _runtime‚2251.058058801 % _timestamp‚1.7844950944565456e+09 _step‚275î< ¦ † /loss/FRt2/detection_loss‚0.5634197435934789 .loss/FBt2/detection_loss‚0.592439427614158 /loss/FRt2/assignment_loss‚2.336273281736456 2loss/FBt1/assignment_loss‚0.012999611231874687 1loss/FBt2/assignment_loss‚0.10006166384931507 1loss/SRqqt1/detection_loss‚0.6799379230915791 /loss/FBt1/detection_loss‚0.8489206876610315 trainer/global_step‚3349 _runtime‚2251.058058801 % _timestamp‚1.7844950944565456e+09 _step‚275 &loss/total_loss‚10.318547403860794 epoch‚8 0loss/FRt1/assignment_loss‚1.1363202394614038 2loss/SRqqt1/assignment_loss‚0.6806658635881052 1loss/SRqqt2/detection_loss‚1.1435652218307935 2loss/SRqqt2/assignment_loss‚0.6831156445777213 /loss/FRt1/detection_loss‚1.5408280956248754“òκ;§ ‚(:° „øôÒ’€ß¾$ disk.nvme1n1.out‚9066.8837890625 cpu‚25.812234183907986 proc.cpu.threads‚17% disk./.usageGB‚1174.0016441345215 disk.rbd0.in‚0& disk.nvme0n1.out‚15998.36083984375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2959.078125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15998.36083984375 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 22822.9140625" disk.dm-1.out‚2121.30419921875$ memory_percent‚98.00195693969727& proc.memory.availableMB‚ 327.359375 network.sent‚ 4.859727e+06 network.recv‚634472& disk.nvme1c1n1.out‚9066.8837890625 disk.md0.out‚ 56.07421875* proc.memory.percent‚18.060779571533203( disk./.usagePercent‚76.4697374456617 disk.nvme0c0n1.in‚0=Xÿ“¨ ‚(:ˆ „øôÒÄþ¿¿ gpu.0.gpu‚92.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚227.406' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚75.802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð«<© ‚(:± “øôÒÑž“¿ proc.cpu.threads‚17 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.out‚23427.76171875 disk.nvme0n1.in‚0% disk.nvme0n1.out‚16396.5439453125 disk.nvme1c1n1.in‚0 cpu‚25.811163692408954 disk.rbd0.out‚0 network.sent‚ 4.922803e+06 network.recv‚638726) disk./.usagePercent‚76.38945300772026 disk.md0.out‚ 56.453125 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1172.7690773010254% disk.nvme1n1.out‚9274.56884765625 disk.nvme0c0n1.in‚0$ memory_percent‚98.00457954406738' proc.memory.availableMB‚ 326.9296875! proc.memory.rssMB‚ 2959.078125* proc.memory.percent‚18.060779571533203" disk.dm-1.out‚2121.52880859375' disk.nvme0c0n1.out‚16396.5439453125' disk.nvme1c1n1.out‚9274.56884765625ÚçäŪ ‚(:’ “øôÒ’ö¾¿ gpu.0.gpu‚76.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚119.0' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.666666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j¹2« - _wandbruntime‚2282  _runtime‚2282Lwlþɬ à _step‚276 ' lr-AdamW/pg1‚0.00015135860079029225 ' lr-AdamW/pg2‚0.00015135860079029225 trainer/global_step‚3399 % _timestamp‚1.7844951254484572e+09 _runtime‚2282.051277174àH™óέ È ' lr-AdamW/pg1‚0.00015135860079029225 ' lr-AdamW/pg2‚0.00015135860079029225 trainer/global_step‚3399 % _timestamp‚1.7844951254484572e+09 _runtime‚2282.051277174 _step‚276”†óÂ2® - _wandbruntime‚2282  _runtime‚2282sœq‘à'¯ ÚO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266299825593856 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2792291116842 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34171123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34171123 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2883422455808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260393960282608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200223971496448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25755474 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18057737 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263499950038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18057741 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197340549040640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26916973 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚266299825593856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263499950038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7697561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263186251399450 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61088094 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26916973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2799981254656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263186147573664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7697733 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2792204330416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260393943243248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚3399 _step‚277 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26916971 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263499932999168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26916971 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197340454930944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2799981254656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25755298 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _runtime‚2282.051364774 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760=úeå'° ßO _runtime‚2282.051364774 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266299825593856 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2792291116842 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34171123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34171123 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2883422455808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260393960282608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200223971496448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25755474 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18057737 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263499950038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18057741 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197340549040640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26916973 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚266299825593856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263499950038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7697561 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263186251399450 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61088094 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26916973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2799981254656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263186147573664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7697733 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2792204330416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260393943243248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚3399 _step‚277 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26916971 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263499932999168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26916971 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197340454930944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2799981254656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25755298 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200•ƒ á2± - _wandbruntime‚2282  _runtime‚2282Š¯wò ² äM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260468233768640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263575082665472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263575099704832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚278 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263575099704832 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61115702 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61105543 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2800766977536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25762682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197396592658944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7699708 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2884231764480 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2793074611554 % _timestamp‚1.7844951258492138e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26924602 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61115702 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34191100 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263261221593656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34180943 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18062793 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26924602 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34191100 D>DeviceStatsMonitor.on_train_batch_end/segment.larg²Âèøe_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26924600 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18062797 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2800678310400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263575082665472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25762501 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263261325419554 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2800766977536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2282.451279863 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266375866682368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200280918533120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2800678310400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266375866682368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2884110716416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7699885 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2792987825016 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200280703375360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266375760975872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34180943 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0‹~Êï&³ éM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26924600 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18062797 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2800678310400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263575082665472 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25762501 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263261325419554 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2800766977536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2282.451279863 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266375866682368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200280918533120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2800678310400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266375866682368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2884110716416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7699885 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2792987825016 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200280703375360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266375760975872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34180943 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260468233768640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263575082665472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263575099704832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚278 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263575099704832 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61115702 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61105543 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2800766977536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25762682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197396592658944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7699708 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2884231764480 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2793074611554 % _timestamp‚1.7844951258492138e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26924602 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61115702 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34191100 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263261221593656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34180943 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18062793 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26924602 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34191100 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0–‚‹>2´ - _wandbruntime‚2282  _runtime‚2282uâóXµ þ trainer/global_step‚3399 % _timestamp‚1.7844951258507857e+09 _runtime‚2282.452342135 /loss/FRt1/assignment_loss‚1.083048266674214 2loss/SRqqt1/assignment_loss‚0.6863348094645577 /loss/FBt2/detection_loss‚0.7899304276611472 _step‚279 1loss/FBt2/assignment_loss‚0.11700621371149575 /loss/FRt2/detection_loss‚0.5837134472436274 &loss/total_loss‚10.628765508213814 /loss/FRt2/assignment_loss‚2.233692333917254 2loss/SRqqt2/assignment_loss‚0.6932307560380513 epoch‚8 0loss/FBt1/assignment_loss‚0.0509607440426134 /loss/FRt1/detection_loss‚1.5758431716551966 1loss/SRqqt1/detection_loss‚0.7655226436035742 0loss/SRqqt2/detection_loss‚1.132073328401255 /loss/FBt1/detection_loss‚0.9174093658008272l>¡Ž ¶ ƒ _step‚279 1loss/FBt2/assignment_loss‚0.11700621371149575 /loss/FRt2/detection_loss‚0.5837134472436274 &loss/total_loss‚10.628765508213814 /loss/FRt2/assignment_loss‚2.233692333917254 2loss/SRqqt2/assignment_loss‚0.6932307560380513 epoch‚8 0loss/FBt1/assignment_loss‚0.0509607440426134 /loss/FRt1/detection_loss‚1.5758431716551966 1loss/SRqqt1/detection_loss‚0.7655226436035742 0loss/SRqqt2/detection_loss‚1.132073328401255 /loss/FBt1/detection_loss‚0.9174093658008272 trainer/global_step‚3399 % _timestamp‚1.7844951258507857e+09 _runtime‚2282.452342135 /loss/FRt1/assignment_loss‚1.083048266674214 2loss/SRqqt1/assignment_loss‚0.6863348094645577 /loss/FBt2/detection_loss‚0.7899304276611472—“DƒrA· ‚(:¶ ¢øôÒ€‚´¿) disk./.usagePercent‚76.38945474703834 disk.rbd0.out‚0* proc.memory.percent‚18.060779571533203 cpu‚25.810002383425335% disk./.usageGB‚1172.7691040039062 disk.md0.in‚0 disk.md0.out‚ 57.08984375 disk.dm-0.in‚0" disk.dm-0.out‚23428.6044921875 disk.nvme0c0n1.in‚0$ memory_percent‚98.00474643707275 proc.cpu.threads‚17% disk.nvme1n1.out‚9275.51318359375" disk.dm-1.out‚2121.68115234375 disk.rbd0.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚16397.9443359375 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 326.90234375! proc.memory.rssMB‚ 2959.078125 network.recv‚642043 disk.nvme1n1.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚16397.9443359375' disk.nvme1c1n1.out‚9275.51318359375 network.sent‚ 4.938563e+06Ý<Ø{“¸ ‚(:ˆ ¢øôÒ¢ƒÀ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚220.809' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.603 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôAP;¹ ‚(:° ±øôÒžãæ¾ disk.nvme1n1.in‚0! disk.dm-1.out‚2121.8154296875 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚16404.64208984375 disk.nvme0n1.in‚0& disk.nvme0n1.out‚16404.64208984375$ memory_percent‚98.00326824188232( proc.memory.availableMB‚ 327.14453125 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 9277.12109375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.8087685271138 proc.cpu.threads‚17 network.sent‚ 4.989876e+06 network.recv‚646545) disk./.usagePercent‚76.38947636427737 disk.md0.in‚0 disk.md0.out‚ 57.39453125 disk.dm-0.in‚0# disk.dm-0.out‚23436.10205078125! proc.memory.rssMB‚ 2959.078125% disk./.usageGB‚1172.7694358825684" disk.nvme1n1.out‚ 9277.12109375* proc.memory.percent‚18.060779571533203aX톓º ‚(:ˆ ±øôÒǺ•¿ gpu.0.gpu‚91.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚224.751' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚74.917 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á‚fê2» - _wandbruntime‚2313  _runtime‚2313w‘%bɼ à ' lr-AdamW/pg1‚0.00015105902403702562 ' lr-AdamW/pg2‚0.00015105902403702562 trainer/global_step‚3449 % _timestamp‚1.7844951569127347e+09 _runtime‚2313.514596314 _step‚280új¹Î½ È ' lr-AdamW/pg1‚0.00015105902403702562 ' lr-AdamW/pg2‚0.00015105902403702562 trainer/global_step‚3449 % _timestamp‚1.7844951569127347e+09 _runtime‚2313.514596314 _step‚280˜}Í52¾ - _wandbruntime‚2313  _runtime‚2313¹Ôñ­ß'¿ ÙO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27298423 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 $ _timestamp‚1.784495156913684e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267257433353728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18310537 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2839242534912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2831529647650 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7804783 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7804618 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2923843716096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2839331194880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203071402309120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2839331194880 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270096658849280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200147341330944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264108469512848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34674118 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34663975 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203071185047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200147435440640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2831442861224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270096658849280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27298421 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26115155 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚165 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34674118 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2923966868480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61962396 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34663975 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266939912374072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2313.515433456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2839242534912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267257416314368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267257433353728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267257416314368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26115324 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _step‚281 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27298421 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264108486552208¡ÂÄõÀ ÞO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚270096764548608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61972541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2839331194880 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270096658849280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200147341330944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264108469512848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34674118 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34663975 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203071185047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200147435440640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2831442861224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270096658849280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27298421 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26115155 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚165 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34674118 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2923966868480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61962396 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34663975 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266939912374072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2313.515433456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2839242534912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267257416314368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267257433353728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267257416314368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26115324 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 _step‚281 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27298421 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264108486552208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27298423 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 $ _timestamp‚1.784495156913684e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267257433353728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18310537 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2839242534912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2831529647650 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7804783 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7804618 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2923843716096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2839331194880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203071402309120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61990004 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34673795 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34673795 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203128129731584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27306052 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34683952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264182760038240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200203573168640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2924650672640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2840028250624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267332565980672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61990004 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61979845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26122397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18315597 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267332583020032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27306050 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200203479058944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34683952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61979845 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2840028250624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7806988 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267015090219962 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7844951573154345e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267332565980672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61990004 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61979845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26122397 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18315597 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267332583020032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27306050 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200203479058944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34683952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61979845 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2840028250624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7806988 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267015090219962 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7844951573154345e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61990004 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34673795 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34673795 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203128129731584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27306052 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34683952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264182760038240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200203573168640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2924650672640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812šõ76,2Ä - _wandbruntime‚2313  _runtime‚23132‹†{Å  epoch‚8 /loss/FRt2/assignment_loss‚2.203827713506738 2loss/SRqqt1/assignment_loss‚0.6151110330540818 2loss/SRqqt2/assignment_loss‚0.6869818351998114 2loss/FBt1/assignment_loss‚0.019520590091378884 /loss/FRt2/detection_loss‚0.5351309275650209 /loss/FBt2/detection_loss‚0.9443884604918787 % _timestamp‚1.7844951573163898e+09 .loss/FRt1/detection_loss‚1.513167589835962 _runtime‚2313.917917263 _step‚283 1loss/SRqqt2/detection_loss‚1.0565727775010476 &loss/total_loss‚10.083402706299184 0loss/FRt1/assignment_loss‚0.9093220317222941 1loss/FBt2/assignment_loss‚0.13722904548881035 1loss/SRqqt1/detection_loss‚0.7000708609795142 /loss/FBt1/detection_loss‚0.7620798408626456 trainer/global_step‚3449{ã÷¼ Æ † 0loss/FRt1/assignment_loss‚0.9093220317222941 1loss/FBt2/assignment_loss‚0.13722904548881035 1loss/SRqqt1/detection_loss‚0.7000708609795142 /loss/FBt1/detection_loss‚0.7620798408626456 trainer/global_step‚3449 epoch‚8 /loss/FRt2/assignment_loss‚2.203827713506738 2loss/SRqqt1/assignment_loss‚0.6151110330540818 2loss/SRqqt2/assignment_loss‚0.6869818351998114 2loss/FBt1/assignment_loss‚0.019520590091378884 /loss/FRt2/detection_loss‚0.5351309275650209 /loss/FBt2/detection_loss‚0.9443884604918787 % _timestamp‚1.7844951573163898e+09 .loss/FRt1/detection_loss‚1.513167589835962 _runtime‚2313.917917263 _step‚283 1loss/SRqqt2/detection_loss‚1.0565727775010476 &loss/total_loss‚10.083402706299184›cÆ“ 7Ç ‚(:¬ ÀøôҌǎ¿ disk.rbd0.out‚0& disk.nvme0n1.out‚16405.45068359375& proc.memory.availableMB‚ 327.109375 cpu‚25.80752931598835 proc.cpu.threads‚17 disk.md0.in‚0# disk.dm-0.out‚23436.77001953125$ memory_percent‚98.00348281860352* proc.memory.percent‚18.060779571533203 network.sent‚ 5.005683e+06 network.recv‚649996 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚16405.45068359375 disk.nvme1n1.in‚0 disk.md0.out‚57.65625 disk.dm-0.in‚0! disk.dm-1.out‚2121.9013671875 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 9277.64453125" disk.nvme1n1.out‚ 9277.64453125 disk.dm-1.in‚0! proc.memory.rssMB‚ 2959.078125) disk./.usagePercent‚76.37401482042128% disk./.usageGB‚1172.5320625305176 disk.rbd0.in‚0Ñ9È ‚(:’ ÀøôÒ“ɽ¿ gpu.0.gpu‚18.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚220.318' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.43933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|²ò§É ‚(:¯ ÏøôÒ™˜æ¾( disk./.usagePercent‚76.3740110933111 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚2122.099609375( proc.memory.availableMB‚ 326.86328125* proc.memory.percent‚18.060779571533203 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.dm-0.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚16410.3017578125& disk.nvme1c1n1.out‚9278.4130859375 disk.nvme1n1.in‚0m.)“$ disk.nvme1n1.out‚9278.4130859375! proc.memory.rssMB‚ 2959.078125 disk.nvme1c1n1.in‚0$ memory_percent‚98.00498485565186 cpu‚25.806320301660207 disk.rbd0.out‚0' disk.nvme0c0n1.out‚16410.3017578125 network.sent‚ 5.056957e+06 network.recv‚654491% disk./.usageGB‚1172.5320053100586 disk.md0.in‚0 disk.md0.out‚ 57.97265625 disk.dm-0.out‚23441.51171875–[ÓžÊ ‚(:“ ÏøôÒƒÔ”¿ gpu.0.gpu‚54.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.564' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.52133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ÛÏ2Ë - _wandbruntime‚2344  _runtime‚2344yG ÉÌ Ã trainer/global_step‚3499 % _timestamp‚1.7844951880269346e+09 _runtime‚2344.628790214 _step‚284 ' lr-AdamW/pg1‚0.00015075489775440604 ' lr-AdamW/pg2‚0.00015075489775440604N̉ÇÎÍ È _step‚284 ' lr-AdamW/pg1‚0.00015075489775440604 ' lr-AdamW/pg2‚0.00015075489775440604 trainer/global_step‚3499 % _timestamp‚1.7844951880269346e+09 _runtime‚2344.628790214œÜBdÇ2Î - _wandbruntime‚2344  _runtime‚2344|dà'Ï ÚO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273893597804032 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27679873 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2344.629558925 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2878681135104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27679871 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2870681392032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27679871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2878592475136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273893492104704 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205918549578752 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270693677174480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7844951880278594e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271014916668928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚161 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273893492104704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267823012821808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18563341 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26474364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18563337 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚157 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202954321840640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _step‚285 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270693781000266 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26474203 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2878681135104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27679873 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62836698 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62846843 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202954227730944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130’'‘å'Ð ßO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18563341 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26474364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18563337 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚157 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202954321840640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _step‚285 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270693781000266 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26474203 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2878681135104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27679873 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62836698 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62846843 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202954227730944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273893597804032 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27679873 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2344.629558925 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2878681135104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27679871 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2870681392032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27679871 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2878592475136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273893492104704 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205918549578752 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270693677174480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7844951880278594e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271014916668928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚161 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273893492104704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267823012821808”96ö2Ñ - _wandbruntime‚2345  _runtime‚2345h°è&Ò âM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2965258527744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267897286307840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26481437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27687500 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62864306 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203010459568640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26481605 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62854147 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35166647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62864306 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _runtime‚ 2345.03263916 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27687500 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 _step‚286 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273969533193216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879378190848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271090049295872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35176804 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27687502 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879378190848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871551673170 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35166647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚164 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205975718096384 $ _timestamp‚1.784495188430681e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2965129091072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271090049295872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879466857984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62854147 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879466857984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271090066335232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2871464886632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273969427486720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898RÅ=T¨Ó çM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271090049295872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879466857984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62854147 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_deviceyÄ·E#_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879466857984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271090066335232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2871464886632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273969427486720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2965258527744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267897286307840 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26481437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27687500 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62864306 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203010459568640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26481605 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62854147 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35166647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62864306 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _runtime‚ 2345.03263916 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27687500 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 _step‚286 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273969533193216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879378190848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271090049295872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35176804 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27687502 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879378190848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871551673170 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35166647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚164 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205975718096384 $ _timestamp‚1.784495188430681e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2965129091072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ž )2Ô - _wandbruntime‚2345  _runtime‚2345pl¯Õ ý &loss/total_loss‚10.800718408612944 _step‚287 1loss/FBt2/assignment_loss‚0.40916660443155595 .loss/FRt2/detection_loss‚0.615943257531776 -loss/FBt2/detection_loss‚1.02934076512937 % _timestamp‚1.7844951884316099e+09 0loss/FRt2/assignment_loss‚2.2596543939233578 /loss/FBt1/detection_loss‚0.9423624394681263 trainer/global_step‚3499 0loss/FRt1/assignment_loss‚0.9000968780601397 2loss/SRqqt1/assignment_loss‚0.6664034121096347 -loss/FRt1/detection_loss‚1.49834064600509 epoch‚8 _runtime‚ 2345.03311422 2loss/SRqqt2/assignment_loss‚0.7534577955921122 2loss/FBt1/assignment_loss‚0.017990825889981306 1loss/SRqqt1/detection_loss‚0.6763178979852126 1loss/SRqqt2/detection_loss‚1.0316434924865865Öé±ÆÖ ‚ 2loss/SRqqt1/assignment_loss‚0.6664034121096347 -loss/FRt1/detection_loss‚1.49834064600509 epoch‚8 _runtime‚ 2345.03311422 2loss/SRqqt2/assignment_loss‚0.7534577955921122 2loss/FBt1/assignment_loss‚0.017990825889981306 1loss/SRqqt1/detection_loss‚0.6763178979852126 1loss/SRqqt2/detection_loss‚1.0316434924865865 &loss/total_loss‚10.800718408612944 _step‚287 1loss/FBt2/assignment_loss‚0.40916660443155595 .loss/FRt2/detection_loss‚0.615943257531776 -loss/FBt2/detection_loss‚1.02934076512937 % _timestamp‚1.7844951884316099e+09 0loss/FRt2/assignment_loss‚2.2596543939233578 /loss/FBt1/detection_loss‚0.9423624394681263 trainer/global_step‚3499 0loss/FRt1/assignment_loss‚0.9000968780601397Ÿ)!¾º+× ‚(:  ÞøôÒÌ°ª¿ disk.md0.in‚0 disk.md0.out‚ 58.62109375 disk.dm-0.out‚23447.02734375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚16416.392578125 network.sent‚ 5.071805e+06) disk./.usagePercent‚76.37399991198056 disk.dm-0.in‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚16416.392578125 disk.nvme1c1n1.in‚0 disk.nvme1c1n1.out‚ 9279.3125! proc.memory.rssMB‚ 2959.078125 disk.nvme1n1.out‚ 9279.3125* proc.memory.percent‚18.060779571533203 proc.cpu.threads‚17% disk./.usageGB‚1172.5318336486816 disk.dm-1.out‚2122.220703125 disk.rbd0.out‚0 disk.nvme1n1.in‚0# memory_percent‚98.0039119720459' proc.memory.availableMB‚ 327.0390625 cpu‚25.805001716146354 network.recv‚656954Ë̾xžØ ‚(:“ ÞøôÒ¼ßÕ¿ gpu.0.gpu‚92.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚228.604' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.20133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õâò3Ù ‚(:¨ íøôÒý®ç¾" disk.dm-1.out‚2122.49365234375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚9303.86865234375 disk.nvme1n1.in‚0* proc.memory.percent‚18.060779571533203 network.recv‚661155$ disk./.usageGB‚1172.769733428955 disk.md0.out‚59.0625% disk.nvme0c0n1.out‚16636.55078125# disk.nvme0n1.out‚16636.55078125 disk.nvme1c1n1.in‚0! proc.memory.rssMB‚ 2959.078125 cpu‚25.803933426702393 disk.nvme0c0n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚98.00395965576172) disk./.usagePercent‚76.38949574525029 disk.nvme0n1.in‚0% disk.nvme1n1.out‚9303.86865234375% proc.memory.availableMB‚ 327.03125 proc.cpu.threads‚17 network.sent‚ 5.121214e+06 disk.md0.in‚0 disk.dm-0.out‚23690.53515625É5®Ú ‚(:’ íøôÒÍ謿 gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.764' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.254666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Áƒô2Û - _wandbruntime‚2376  _runtime‚2376¯u€ÉÜ Ã trainer/global_step‚3549 % _timestamp‚1.7844952196532667e+09 _runtime‚2376.255088035 _step‚288 ' lr-AdamW/pg1‚0.00015044624147589632 ' lr-AdamW/pg2‚0.00015044624147589632$¿1VÎÝ È _step‚288 ' lr-AdamW/pg1‚0.00015044624147589632 ' lr-AdamW/pg2‚0.00015044624147589632 trainer/global_step‚3549 % _timestamp‚1.7844952196532667e+09 _runtime‚2376.255088035 !X(+2Þ - _wandbruntime‚2376  _runtime‚2376Z0èà'ß ÚO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205761114130944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274447545800674 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 % _timestamp‚1.7844952196542225e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚164 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274772399984128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205761208240640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271537522052048 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63711000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2918031075328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2917942415360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18816137 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274447441974888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2910006709266 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚277690431059456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 _runtime‚2376.256014197 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28061323 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28061321 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277690325360128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚289 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8016830 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277690431059456 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3004966918144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2918031075328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35649679 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18816141 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2917942415360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28061321 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35659822 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35649679 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35659822 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277690325360128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208765951605248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274772399984128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28061323 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63721145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274772399984128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205761208240640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271537522052048 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63711000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2918031075328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2917942415360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18816137 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274447441974888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2910006709266 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚277690431059456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 _runtime‚2376.256014197 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28061323 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28061321 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277690325360128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚289 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8016830 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277690431059456 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3004966918144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2918031075328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35649679 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18816141 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2917942415360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28061321 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35659822 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35649679 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35659822 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277690325360128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/allocap3\=ted_bytes.large_pool.peak‚ 12065026048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208765951605248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274772399984128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28061323 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63721145 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277766260742144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274522515994880 _step‚290 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2910703417440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3005771008512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208822906013696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚2376.657309128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26840154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274847532611072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274847532611072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35659499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28068950 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35659499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚208823116977152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28068952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26840308 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18821197 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18821193 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚3549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274847549650432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277766366448640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2918728131072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8018961 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35669656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8019111 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63738608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205817345968640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205817251858944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63728449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63728449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚150 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277766260742144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274522515994880 _step‚290 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2910703417440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3005771008512 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208822906013696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚2376.657309128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26840154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274847532611072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274847532611072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35659499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28068950 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35659499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚208823116977152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28068952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26840308 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18821197 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18821193 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚3549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274847549650432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277766366448640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2918728131072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8018961 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35669656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8019111 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63738608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205817345968640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205817251858944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63728449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63728449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚150 B2ë - _wandbruntime‚2408  _runtime‚2408âõó>Éì à trainer/global_step‚3599 % _timestamp‚1.7844952519149935e+09 _runtime‚2408.516839067 _step‚292 ' lr-AdamW/pg1‚0.00015013307502591245 ' lr-AdamW/pg2‚0.00015013307502591245ë€JøÎí È trainer/global_step‚3599 % _timestamp‚1.7844952519149935e+09 _runtime‚2408.516839067 _step‚292 ' lr-AdamW/pg1‚0.00015013307502591245 ' lr-AdamW/pg2‚0.00015013307502591245¤Ýdéá2î - _wandbruntime‚2408  _runtime‚2408Ý¥[ï ÚO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2957381015552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27191242 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28442771 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281487264314880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281487158615552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36152674 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64585302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281487264314880 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8122301 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281487158615552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36142531 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64595447 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19068941 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _step‚293 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚278529866259968 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28442773 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211613557206528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278201206775296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278529866259968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2957381015552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19068937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211613331555840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3045462565888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36152674 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275252065361008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0+·©Hå'ð ßO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275252065361008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2957381015552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27191242 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28442771 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281487264314880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281487158615552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36152674 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64585302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281487264314880 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8122301 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281487158615552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36142531 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64595447 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19068941 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _step‚293 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚278529866259968 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28442773 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211613557206528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278201206775296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278529866259968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2957381015552 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19068937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211613331555840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3045462565888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36152674 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248¥EÛÂ2ñ - _wandbruntime‚2408  _runtime‚2408ë‚Kê&ò äM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281563199704064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28450400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36162508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27198404 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64612910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _runtime‚2408.919814297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3046263379456 % _timestamp‚1.7844952523178008e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28450402 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 _step‚294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2950028734786 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2958166738432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281563093997568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64602751 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64602751 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275326355886400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2958166738432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19073997 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278276384621186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211670280590336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278605032965632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27198230 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8124237 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36152351 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278605015926272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279<;2W#ó éM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278605032965632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27198230 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8124237 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36152351 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278605015926272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281563199704064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28450400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36162508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27198404 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64612910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _runtime‚2408.919814297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3046263379456 % _timestamp‚1.7844952523178008e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28450402 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 _step‚294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2950028734786 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2958166738432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281563093997568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64602751 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64602751 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275326355886400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2958166738432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_b\«¶¢˜atch_end/inactive_split.large_pool.allocated‚19073997 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278276384621186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211670280590336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¦Ùp2ô - _wandbruntime‚2408  _runtime‚2408¹à+õ ÿ 1loss/SRqqt2/assignment_loss‚0.728217659353399 2loss/FBt1/assignment_loss‚0.023036025898215726 /loss/FRt2/detection_loss‚0.6109654072377182 1loss/SRqqt1/detection_loss‚0.6947710092852579 trainer/global_step‚3599 _step‚295 /loss/FRt1/assignment_loss‚1.018498588142804 /loss/FRt2/assignment_loss‚2.231654697994016 2loss/SRqqt1/assignment_loss‚0.6048173200306244 1loss/SRqqt2/detection_loss‚1.1009324179949607 /loss/FBt1/detection_loss‚0.6985052534243018 .loss/FBt2/detection_loss‚0.830623349905405 1loss/FBt2/assignment_loss‚0.14283258995620557 &loss/total_loss‚10.130987469036334 epoch‚8 /loss/FRt1/detection_loss‚1.4461331498134267 % _timestamp‚1.7844952523194108e+09 _runtime‚2408.920972969d$Z£ ö „ /loss/FRt1/assignment_loss‚1.018498588142804 /loss/FRt2/assignment_loss‚2.231654697994016 2loss/SRqqt1/assignment_loss‚0.6048173200306244 1loss/SRqqt2/detection_loss‚1.1009324179949607 /loss/FBt1/detection_loss‚0.6985052534243018 .loss/FBt2/detection_loss‚0.830623349905405 1loss/FBt2/assignment_loss‚0.14283258995620557 &loss/total_loss‚10.130987469036334 epoch‚8 /loss/FRt1/detection_loss‚1.4461331498134267 % _timestamp‚1.7844952523194108e+09 _runtime‚2408.920972969 1loss/SRqqt2/assignment_loss‚0.728217659353399 2loss/FBt1/assignment_loss‚0.023036025898215726 /loss/FRt2/detection_loss‚0.6109654072377182 1loss/SRqqt1/detection_loss‚0.6947710092852579 trainer/global_step‚3599 _step‚295§Lãcõ4÷ ‚(:© šùôҷ𻿠proc.cpu.threads‚17% disk./.usageGB‚1172.7525825500488& disk.nvme1c1n1.out‚9308.8662109375 disk.md0.in‚0$ memory_percent‚98.00586700439453* proc.memory.percent‚18.060779571533203 cpu‚25.815785399383458 network.sent‚ 5.207103e+06 network.recv‚672236 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0( disk./.usagePercent‚76.3883786060934 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0# disk.nvme0n1.out‚16657.26171875% proc.memory.availableMB‚ 326.71875! proc.memory.rssMB‚ 2959.078125 disk.dm-1.out‚2122.873046875% disk.nvme0c0n1.out‚16657.26171875 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚9308.8662109375 disk.md0.out‚ 60.2421875" disk.dm-0.out‚23713.3154296875†±œø ‚(:’ šùôÒ´¨è¿ gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚223.135' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.37833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„S2ù - _wandbruntime‚2422  _runtime‚2422¼x[—=ú ‘{ 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2192942500114441 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6287304759025574 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02691327966749668 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7425909042358398 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7119439244270325 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17109839618206024 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8050724267959595 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8347713947296143 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6070480346679688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.06846843659877777 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8506047129631042 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3333333134651184 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20751050114631653 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4958851635456085 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6022043228149414 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7724184989929199 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20911511778831482 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2121983915567398 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6577250361442566 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14007951319217682 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47099167108535767 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8050724267959595 *jet/accuracy_1_of_3‚0.9116910099983215 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 particle/accuracy_3_of_5‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03896864876151085 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6068313121795654 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7722089886665344 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09083059430122375 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904621839523315 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27458587288856506 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7270569801330566 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9877341985702515 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07927580922842026 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9685618877410889 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967077374458313 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8873338103294373 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6551340818405151 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977338910102844 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9614708423614502 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9422012567520142 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06656238436698914 +particle/specificity‚0.9047513604164124 7validation_average_jet_accuracy‚0.35461917519569397 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16810807585716248 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6782069206237793 ,jet/accuracy_4_of_4‚0.007130124606192112 'particle/f_score‚0.6315068602561951 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6202467679977417 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17957337200641632 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12781402468681335 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46361956000328064 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8258126378059387 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.652300238609314 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6736733913421631 *jet/accuracy_2_of_4‚0.7796689867973328 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +jet/accuracy_1_of_1‚0.31311336159706116 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17863993346691132 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8469313383102417 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9895926713943481 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7013558745384216 )jet/accuracy_2_of_3‚0.510673999786377 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8409675359725952 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7837491035461426 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36036431789398193 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9711006879806519 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6090410351753235 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968254566192627 *jet/accuracy_1_of_2‚0.7642560601234436 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07628883421421051 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9894558191299438 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 trainer/global_step‚3599 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3591054379940033 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07568195462226868 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5831272006034851 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5635016560554504 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6522961854934692 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9802750945091248 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6530717015266418 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.35384848713874817 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033117275685071945 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.807692289352417 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.892313539981842 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 /particle/accuracy_3_of_4‚0.9914682507514954 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7040175199508667 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 (particle/accuracy‚0.8440716862678528 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5544353127479553 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19188973307609558 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5989392995834351 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6007876992225647 'loss/val_total_loss‚9.6176450218244 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21905453503131866 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6761350631713867 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020421475172042847 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3240740895271301 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20751050114631653 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 _runtime‚ 2422.2449217 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19240370392799377 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5616611242294312 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11163464188575745 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_2_of_2‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.300496369600296 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3982723355293274 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06656238436698914 +particle/sensitivity‚0.6221727728843689 _step‚296 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1605471521615982 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8925586938858032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 % _timestamp‚1.7844952656426861e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15756632387638092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5977036356925964 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.36420178413391113 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08036218583583832 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5822333693504333 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05194522812962532 particle/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9875285029411316 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.841458797454834 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12351406365633011 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7095383405685425 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9989582896232605 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919066429138184 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1890743523836136 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1947767585515976 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26388904452323914 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3240740895271301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9775175452232361 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6044428944587708 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2201855331659317 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6358002424240112 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9442216753959656 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2002561390399933 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961692094802856 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6546361446380615 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8379629254341125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7188963890075684 epoch‚8 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.13611271977424622 *jet/accuracy_1_of_4‚0.9679206013679504 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9850054383277893 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6326998472213745 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4650075137615204 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18777921795845032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5095833539962769 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7450006604194641 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20603232085704803 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6090410351753235 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27063336968421936 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043315742164850235 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15774981677532196 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6018196940422058 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6507065296173096 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375¿!)"N3û –{ =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08036218583583832 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5822333693504333 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05194522812962532 particle/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9875285029411316 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.841458797454834 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12351406365633011 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7095383405685425 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9989582896232605 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919066429138184 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1890743523836136 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1947767585515976 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26388904452323914 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38461539149284363 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3240740895271301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9775175452232361 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6044428944587708 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2201855331659317 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6358002424240112 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9442216753959656 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2002561390399933 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961692094802856 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6546361446380615 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8379629254341125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7188963890075684 epoch‚8 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.13611271977424622 *jet/accuracy_1_of_4‚0.9679206013679504 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9850054383277893 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6326998472213745 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4650075137615204 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18777921795845032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5095833539962769 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7450006604194641 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20603232085704803 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6090410351753235 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27063336968421936 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043315742164850235 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15774981677532196 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6018196940422058 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6507065296173096 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2192942500114441 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6287304759025574 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02691327966749668 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7425909042358398 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7119439244270325 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17109839618206024 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8050724267959595 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8347713947296143 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6070480346679688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.06846843659877777 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8506047129631042 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3333333134651184 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20751050114631653 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4958851635456085 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6022043228149414 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7724184989929199 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20911511778831482 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2121983915567398 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44675928354263306 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6577250361442566 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14007951319217682 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47099167108535767 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8050724267959595 *jet/accuracy_1_of_3‚0.9116910099983215 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 particle/accuracy_3_of_5‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03896864876151085 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6068313121795654 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7722089886665344 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09083059430122375 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904621839523315 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27458587288856506 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7270569801330566 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9877341985702515 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07927580922842026 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9685618877410889 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967077374458313 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8873338103294373 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6551340818405151 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977338910102844 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9614708423614502 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9422012567520142 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06656238436698914 +particle/specificity‚0.9047513604164124 7validation_average_jet_accuracy‚0.35461917519569397 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16810807585716248 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6782069206237793 ,jet/accuracy_4_of_4‚0.007130124606192112 'particle/f_score‚0.6315068602561951 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6202467679977417 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17957337200641632 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12781402468681335 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46361956000328064 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8258126378059387 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.652300238609314 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6736733913421631 *jet/accuracy_2_of_4‚0.7796689867973328 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +jet/accuracy_1_of_1‚0.31311336159706116 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17863993346691132 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8469313383102417 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9895926713943481 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7013558745384216 )jet/accuracy_2_of_3‚0.510673999786377 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8409675359725952 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7837491035461426 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36036431789398193 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9711006879806519 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6090410351753235 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968254566192627 *jet/accuracy_1_of_2‚0.7642560601234436 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07628883421421051 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9894558191299438 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 trainer/global_step‚3599 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3591054379940033 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07568195462226868 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5831272006034851 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5635016560554504 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6522961854934692 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9802750945091248 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6530717015266418 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.35384848713874817 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.38461539149284363 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033117275685071945 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8076?ž!ØN 92289352417 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.892313539981842 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 /particle/accuracy_3_of_4‚0.9914682507514954 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7040175199508667 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 (particle/accuracy‚0.8440716862678528 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5544353127479553 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19188973307609558 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5989392995834351 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6007876992225647 'loss/val_total_loss‚9.6176450218244 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21905453503131866 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6761350631713867 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020421475172042847 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3240740895271301 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20751050114631653 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 _runtime‚ 2422.2449217 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19240370392799377 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5616611242294312 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11163464188575745 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 particle/accuracy_2_of_2‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.300496369600296 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3982723355293274 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06656238436698914 +particle/sensitivity‚0.6221727728843689 _step‚296 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1605471521615982 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8925586938858032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 % _timestamp‚1.7844952656426861e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15756632387638092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5977036356925964 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.36420178413391113¨èÉÕ0ü ‚(:¥ ©ùôÒ÷Ÿê¾$ memory_percent‚98.03886413574219* proc.memory.percent‚18.093490600585938 cpu‚25.823988581822196 proc.cpu.threads‚17) disk./.usagePercent‚76.38839798706631% disk./.usageGB‚1172.7528800964355 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0$ disk.nvme1c1n1.out‚ 9373.23046875 proc.memory.rssMB‚ 2964.4375 disk.md0.in‚0 disk.dm-1.out‚2186.794921875$ proc.memory.availableMB‚321.3125 network.sent‚ 5.255739e+06 network.recv‚676807 disk.nvme1n1.in‚0 disk.md0.out‚60.40625 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚16658.0712890625" disk.nvme1n1.out‚ 9373.23046875" disk.dm-0.out‚23714.1904296875% disk.nvme0n1.out‚16658.0712890625 disk.nvme1c1n1.in‚0:gbžý ‚(:“ ©ùôÒí¬Æ¿ gpu.0.gpu‚91.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.698' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.23266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ ²88þ ‚(:­ ¸ùôÒ‘£Ñ¾# disk.dm-0.out‚23721.28662109375$ memory_percent‚98.03907871246338( proc.memory.availableMB‚ 321.27734375 proc.memory.rssMB‚ 2964.4375 cpu‚25.822626505817937 network.recv‚680777& disk.nvme0n1.out‚16665.22607421875# disk.nvme1c1n1.out‚ 9410.2109375 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 5.313622e+06 disk.md0.in‚0* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17% disk./.usageGB‚1172.8146667480469 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0! disk.nvme1n1.out‚ 9410.2109375 disk.md0.out‚ 60.82421875 disk.dm-1.out‚2222.912109375 disk.rbd0.in‚0) disk./.usagePercent‚76.39242252063487( disk.nvme0c0n1.out‚16665.22607421875 disk.dm-0.in‚08›jÿ ‚(:’ ¸ùôÒœ¬”¿ gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚215.558' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.85266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S&x#2€- _wandbruntime‚2453  _runtime‚24534‡ëÈ ' lr-AdamW/pg1‚0.00014981541851855036 ' lr-AdamW/pg2‚0.00014981541851855036 trainer/global_step‚3649 % _timestamp‚1.7844952973286715e+09 _runtime‚ 2453.93107716 _step‚297y7ëZÍ‚Ç _runtime‚ 2453.93107716 _step‚297 ' lr-AdamW/pg1‚0.00014981541851855036 ' lr-AdamW/pg2‚0.00014981541851855036 trainer/global_step‚3649 % _timestamp‚1.7844952973286715e+09©'Qái2ƒ- _wandbruntime‚2453  _runtime‚2453N_Ñ]ß'„ÙO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279940041627440 _runtime‚ 2453.93114733 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3012160332800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28946021 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283283853910528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36717890 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212178346245632 % _timestamp‚1.7844952973293326e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36728033 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3101401398272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286296014243328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286295908544000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3012160332800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3012071672832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19429996 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3003899029448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19429992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283283836871168 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3003812243022 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286295908544000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27706643 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212178252135936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65663911 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27706466 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283283836871168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283283853910528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8276647 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282943957696248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946023 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3012071672832 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283283853910528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36717890 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212178346245632 % _timestamp‚1.7844952973293326e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36728033 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3101401398272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286296014243328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286295908544000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3012160332800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3012071672832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19429996 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3003899029448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19429992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283283836871168 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3003812243022 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286295908544000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27706643 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212178252135936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65663911 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27706466 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283283836871168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283283853910528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8276647 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282943957696248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946023 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3012071672832 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65691519 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283018927890454 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3004682524160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28953652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3012946055680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3102217169408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36727710 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286371949632512 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3012946055680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3012857388544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36737867 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3102336120320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27713866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19435048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65681360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3004595737622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36737867 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28953650 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28953650 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283359003576832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283359003576832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36727710 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212234483973632 % _timestamp‚1.7844952977315223e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286371949632512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283358986537472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65691519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212234389863936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.™Hýé$ small_pool.current‚ 118950912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2454.333504103 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286371843926016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3012857388544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65681360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215336607033344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚3649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283019031716352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280014349192192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8278814 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280014332152832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27713688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8278640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200Å› ï&ˆéM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65681360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3004595737622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36737867 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28953650 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28953650 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283359003576832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283359003576832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36727710 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212234483973632 % _timestamp‚1.7844952977315223e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286371949632512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283358986537472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65691519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212234389863936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2454.333504103 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286371843926016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3012857388544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65681360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215336607033344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚3649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283019031716352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280014349192192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8278814 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280014332152832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27713688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8278640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215336820093952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65691519 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283018927890454 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3004682524160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28953652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3012946055680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3102217169408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36727710 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286371949632512 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3012946055680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3012857388544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36737867 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3102336120320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27713866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19435048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159«nWçr2‰- _wandbruntime‚2454  _runtime‚2454NáôŠ€ epoch‚9 _runtime‚2454.333995194 _step‚300 1loss/SRqqt1/detection_loss‚0.6365655425695896 1loss/SRqqt2/detection_loss‚1.0654831278445167 /loss/FBt1/detection_loss‚0.6399085328341103 /loss/FBt2/detection_loss‚0.7653788885991297 %loss/total_loss‚9.949348791329633 trainer/global_step‚3649 $ _timestamp‚1.784495297732481e+09 0loss/FRt1/assignment_loss‚1.0630922648085266 0loss/FRt2/assignment_loss‚2.2075754014469773 2loss/SRqqt1/assignment_loss‚0.6230159745036453 2loss/SRqqt2/assignment_loss‚0.6208345987990824 1loss/FBt1/assignment_loss‚0.08277399590036526 1loss/FBt2/assignment_loss‚0.12544292857763087 /loss/FRt1/detection_loss‚1.5467354925447183 /loss/FRt2/detection_loss‚0.5725420429013425Fô›Ä ‹… /loss/FRt2/detection_loss‚0.5725420429013425 epoch‚9 _runtime‚2454.333995194 _step‚300 1loss/SRqqt1/detection_loss‚0.6365655425695896 1loss/SRqqt2/detection_loss‚1.0654831278445167 /loss/FBt1/detection_loss‚0.6399085328341103 /loss/FBt2/detection_loss‚0.7653788885991297 %loss/total_loss‚9.949348791329633 trainer/global_step‚3649 $ _timestamp‚1.784495297732481e+09 0loss/FRt1/assignment_loss‚1.0630922648085266 0loss/FRt2/assignment_loss‚2.2075754014469773 2loss/SRqqt1/assignment_loss‚0.6230159745036453 2loss/SRqqt2/assignment_loss‚0.6208345987990824 1loss/FBt1/assignment_loss‚0.08277399590036526 1loss/FBt2/assignment_loss‚0.12544292857763087 /loss/FRt1/detection_loss‚1.5467354925447183¬gS9Œ‚(:® ÇùôÒ¿›¿ disk.dm-1.in‚0" disk.dm-1.out‚2250.84912109375 disk.rbd0.in‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 321.24609375 network.recv‚684412 disk.md0.in‚0 disk.rbd0.out‚0# disk.nvme1n1.out‚9438.478515625 cpu‚25.821061853075815 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚16666.466796875) disk./.usagePercent‚76.39242152673883 disk.dm-0.in‚0% disk.nvme1c1n1.out‚9438.478515625 disk.nvme1n1.in‚0$ memory_percent‚98.03926944732666 proc.memory.rssMB‚ 2964.4375% disk./.usageGB‚1172.8146514892578$ disk.nvme0n1.out‚16666.466796875 disk.nvme1c1n1.in‚0* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17 network.sent‚ 5.331241e+06 disk.md0.out‚ 61.0234375# disk.dm-0.out‚23722.42333984375—Î&tž‚(:“ ÇùôÒèÖο gpu.0.gpu‚89.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚223.819' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.60633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¡zz:Ž‚(:¯ ÖùôÒú¿ç¾' disk.nvme1c1n1.out‚9439.38037109375# memory_percent‚98.4323263168335( proc.memory.availableMB‚ 256.84765625* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17! disk.dm-0.out‚23726.771484375 disk.dm-1.in‚0" disk.dm-1.out‚2250.93505859375 network.sent‚ 5.379575e+06 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.nvme1c1n1.in‚0 proc.memory.rssMB‚ 2964.4375 cpu‚25.819637910657146 disk.rbd0.out‚0 network.recv‚688260% disk./.usageGB‚1172.8147621154785% disk.nvme1n1.out‚9439.38037109375 disk.md0.out‚ 61.3359375 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚16670.685546875 disk.nvme0n1.in‚0) disk./.usagePercent‚76.39242873248516 disk.rbd0.in‚0$ disk.nvme0n1.out‚16670.685546875šn©Êž‚(:“ ÖùôÒ†—ª¿ gpu.0.gpu‚92.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚118.187' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.39566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð2øË2- _wandbruntime‚2485  _runtime‚2485zf•tÉ‘Ã ' lr-AdamW/pg1‚0.00014949329235629388 ' lr-AdamW/pg2‚0.00014949329235629388 trainer/global_step‚3699 % _timestamp‚1.7844953287197526e+09 _runtime‚2485.321607515 _step‚301ñO(Î’È _step‚301 ' lr-AdamW/pg1‚0.00014949329235629388 ' lr-AdamW/pg2‚0.00014949329235629388 trainer/global_step‚3699 % _timestamp‚1.7844953287197526e+09 _runtime‚2485.321607515­¤Ea2“- _wandbruntime‚2485  _runtime‚2485ŒÄnëà'”ÚO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37210742 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37220885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66548358 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29327473 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19682796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19682792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287041337225728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214985232645632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3699 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3051421613056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287041320186368 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3141988507136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283654567897040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29327473 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290092741799424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚161 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29327471 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28067233 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286697618670870 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283654584936400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3051510273024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28067068 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290092741799424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29327471 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3051510273024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3142113756672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290092847498752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286697722496656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8384437 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218127127043072 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37220885 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218127346402304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚302 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66538213˜Âˆî·•ßO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29327473 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19682796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19682792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287041337225728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214985232645632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3699 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3051421613056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287041320186368 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3141988507136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283654567897040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29327473 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290092741799424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚161 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29327471 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28067233 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286697618670870 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283654584936400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3051510273024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28067068 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290092741799424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29327471 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3051510273024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 314211375667Jr‡r.2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290092847498752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286697722496656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8384437 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218127127043072 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37220885 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218127346402304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚302 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66538213 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37210742 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37220885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66548358 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696®8ÜÊ^2–- _wandbruntime‚2485  _runtime‚2485v›‚ê&—äM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3052207328768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287116486892032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚2485.724545895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66555662 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335102 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37230719 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37220562 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8386321 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28074169 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚173 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283728858422432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335100 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3052295995904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3052295995904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386494 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37230719 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28074346 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142922434560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66565821 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335102 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37220562 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290168782887936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19687848 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286772796516760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 % _timestamp‚1.7844953291225631e+09ûèÞGï&˜éM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3052207328768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287116486892032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚2485.724545895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66555662 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335102 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37230719 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37220562 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8386321 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28074169 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚173 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283728858422432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335100 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3052295995904 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3052295995904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386494 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37230719 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28074346 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142922434560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66565821 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335102 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37220562 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290168782887936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19687848 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286772796516760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 % _timestamp‚1.7844953291225631e+09¯ÝpGå2™- _wandbruntime‚2485  _runtime‚2485!£š€ 0loss/FRt2/assignment_loss‚2.2544066695163125 1loss/SRqqt1/detection_loss‚0.6529223396292984 2loss/SRqqt2/assignment_loss‚0.8133573428394754 1loss/FBt2/assignment_loss‚0.11570294938390874 /loss/FRt1/detection_loss‚1.4967673311678547 1loss/SRqqt2/detection_loss‚1.1132549942276628 %loss/total_loss‚10.23770829409879 _runtime‚2485.724991966 2loss/SRqqt1/assignment_loss‚0.6618800883936267 /loss/FRt2/detection_loss‚0.6095008949807379 /loss/FBt1/detection_loss‚0.5998316130968272 trainer/global_step‚3699 $ _timestamp‚1.784495329123502e+09 _step‚304 0loss/FRt1/assignment_loss‚0.9047627397097987 1loss/FBt1/assignment_loss‚0.08514429212217545 /loss/FBt2/detection_loss‚0.9301770390311122 epoch‚9œäó ›… 1loss/SRqqt1/detection_loss‚0.6529223396292984 2loss/SRqqt2/assignment_loss‚0.8133573428394754 1loss/FBt2/assignment_loss‚0.11570294938390874 /loss/FRt1/detection_loss‚1.4967673311678547 1loss/SRqqt2/detection_loss‚1.1132549942276628 %loss/total_loss‚10.23770829409879 _runtime‚2485.724991966 2loss/SRqqt1/assignment_loss‚0.6618800883936267 /loss/FRt2/detection_loss‚0.6095008949807379 /loss/FBt1/detection_loss‚0.5998316130968272 trainer/global_step‚3699 $ _timestamp‚1.784495329123502e+09 _step‚304 0loss/FRt1/assignment_loss‚0.9047627397097987 1loss/FBt1/assignment_loss‚0.08514429212217545 /loss/FBt2/detection_loss‚0.9301770390311122 epoch‚9 0loss/FRt2/assignment_loss‚2.2544066695163125°7·_’>œ‚(:³ åùôÒ•Üß¿ disk.md0.out‚61.84375# disk.dm-0.out‚23727.84716796875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚16672.17138671875 disk.nvme1n1.in‚0$ memory_percent‚98.43258857727051' proc.memory.availableMB‚ 256.8046875 proc.memory.rssMB‚ 2964.4375 network.sent‚ 5.384345e+06) disk./.usagePercent‚76.39242972638122 disk.dm-0.in‚0' disk.nvme1c1n1.out‚9440.30419921875% disk.nvme1n1.out‚9440.30419921875* proc.memory.percent‚18.093490600585938 cpu‚25.818202230570467 proc.cpu.threads‚17" disk.dm-1.out‚2251.19091796875 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚16672.17138671875 disk.nvme1c1n1.in‚0 network.recv‚691634% disk./.usageGB‚1172.8147773742676 disk.md0.in‚0“ÎoO‚(:’ åùôÒµ‚‰¿ gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚163.388' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚54.46266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o)ÂßAž‚(:¶ ôùôÒÊ‹¿' disk.nvme1c1n1.out‚9441.46923828125 disk.md0.out‚ 62.3671875# disk.dm-0.out‚23730.31103515625 disk.dm-1.in‚0" disk.dm-1.out‚2251.30419921875 proc.cpu.threads‚17) disk./.usagePercent‚76.39244488329592% disk./.usageGB‚1172.8150100708008 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚16674.70263671875 disk.nvme0n1.in‚0 disk.md0.in‚0 network.sent‚ 5.446528e+06& disk.nvme0n1.out‚16674.70263671875 disk.nvme1n1.in‚0 disk.dm-0.in‚0$ memory_percent‚98.43442440032959( proc.memory.availableMB‚ 256.50390625 proc.memory.rssMB‚ 2964.4375 cpu‚25.816817558530005% disk.nvme1n1.out‚9441.46923828125 network.recv‚695673* proc.memory.percent‚18.093490600585938 disk.nvme1c1n1.in‚0Šµ“Ÿ‚(:ˆ ôùôÒдҿ gpu.0.gpu‚53.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚225.609' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚75.203 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00séò2 - _wandbruntime‚2516  _runtime‚2516Èc?AÆ¡À & lr-AdamW/pg1‚0.0001491667172287045 & lr-AdamW/pg2‚0.0001491667172287045 trainer/global_step‚3749 % _timestamp‚1.7844953597777007e+09 _runtime‚ 2516.37955599 _step‚305 ‡FˢŠ_step‚305 & lr-AdamW/pg1‚0.0001491667172287045 & lr-AdamW/pg2‚0.0001491667172287045 trainer/global_step‚3749 % _timestamp‚1.7844953597777007e+09 _runtime‚ 2516.37955599±Dp¸2£- _wandbruntime‚2516  _runtime‚2516ˆ»+p ¤ÚO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 114763776 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293889575054848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293889680754176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19935592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚174 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 % _timestamp‚1.7844953597786481e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290798803501568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29708921 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3090860213248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37713737 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67412515 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 A7DeviceStatsMonitor.on_train_batch_start/actiðc}Npve.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287369111206000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28427816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3090771553280 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29708923 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29708923 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220974789833216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217792024935936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 _runtime‚2516.384286928 _step‚306 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293889575054848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293889680754176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19935592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚174 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 % _timestamp‚1.7844953597786481e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290798803501568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29708921 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3090860213248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37713737 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67412515 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287369111206000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28427816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3090771553280 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29708923 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29708923 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220974789833216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217792024935936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 _runtime‚2516.384286928 _step‚306 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19940652 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217848162663936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293965616143360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290526457491270 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67429964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3083072799238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37713414 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚2516.780714144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293965510436864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290873970207232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3749 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37713414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37723571 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3183364839936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290873970207232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29716550 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287443401731392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290873953167872 % _timestamp‚1.7844953601787155e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28434861 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28435038 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290526561317168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67429964 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8494386 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67440123 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3091557268992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221031738467328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67440123 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29716552X_Mÿl¨éM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚177 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37723571 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3183364839936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290873970207232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29716550 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287443401731392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290873953167872 % _timestamp‚1.7844953601787155e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28434861 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 210963456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28435038 E=Dev{à ƒiceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290526561317168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67429964 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8494386 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67440123 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3091557268992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221031738467328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67440123 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29716552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3091557268992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19940648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19940652 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217848162663936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293965616143360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290526457491270 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67429964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3083072799238 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37713414 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚2516.780714144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293965510436864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290873970207232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116853760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3749 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37713414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0³=1VÜ2©- _wandbruntime‚2516  _runtime‚2516³ûUyª $loss/total_loss‚9.95433582002718 epoch‚9 1loss/SRqqt2/detection_loss‚1.1328098008518255 0loss/FRt1/assignment_loss‚0.7777776278618003 2loss/FBt1/assignment_loss‚0.011642262037454786 /loss/FRt1/detection_loss‚1.5568810857660254 1loss/SRqqt1/detection_loss‚0.6196085439254977 % _timestamp‚1.7844953601796362e+09 0loss/FRt2/assignment_loss‚2.3646930890279894 2loss/SRqqt1/assignment_loss‚0.6145849726317427 2loss/SRqqt2/assignment_loss‚0.7331322274631052 1loss/FBt2/assignment_loss‚0.21723703843040404 /loss/FRt2/detection_loss‚0.5698533933833541 /loss/FBt1/detection_loss‚0.6441986209943275 /loss/FBt2/detection_loss‚0.7119171576536546 _runtime‚2516.781272495 trainer/global_step‚3749 _step‚308d‰f «† epoch‚9 1loss/SRqqt2/detection_loss‚1.1328098008518255 0loss/FRt1/assignment_loss‚0.7777776278618003 2loss/FBt1/assignment_loss‚0.011642262037454786 /loss/FRt1/detection_loss‚1.5568810857660254 1loss/SRqqt1/detection_loss‚0.6196085439254977 % _timestamp‚1.7844953601796362e+09 0loss/FRt2/assignment_loss‚2.3646930890279894 2loss/SRqqt1/assignment_loss‚0.6145849726317427 2loss/SRqqt2/assignment_loss‚0.7331322274631052 1loss/FBt2/assignment_loss‚0.21723703843040404 /loss/FRt2/detection_loss‚0.5698533933833541 /loss/FBt1/detection_loss‚0.6441986209943275 /loss/FBt2/detection_loss‚0.7119171576536546 _runtime‚2516.781272495 trainer/global_step‚3749 _step‚308 $loss/total_loss‚9.95433582002718´²Hxó4¬‚(:© ƒúôÒÄø±¿" disk.nvme1n1.out‚ 9442.85546875 cpu‚25.81552759795104 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme0n1.out‚16676.03173828125$ disk.nvme1c1n1.out‚ 9442.85546875 disk.dm-1.in‚0 network.sent‚ 5.464299e+06 network.recv‚699357% disk./.usageGB‚1172.8156929016113 disk.nvme1c1n1.in‚0 disk.dm-1.out‚ 2251.9140625( proc.memory.availableMB‚ 256.08203125 proc.memory.rssMB‚ 2964.4375* proc.memory.percent‚18.093490600585938 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.md0.out‚62.90625 disk.dm-0.in‚0# disk.dm-0.out‚23731.23876953125$ memory_percent‚98.43699932098389) disk./.usagePercent‚76.39248936014404( disk.nvme0c0n1.out‚16676.03173828125 disk.nvme1n1.in‚0ä‘÷œ­‚(:‘ ƒúôÒôŒà¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚114.817' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.27233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÖÜg4®‚(:© ’úôÒŒüŠ¿ network.recv‚703477 disk.md0.in‚0 disk.md0.out‚ 63.5234375 disk.dm-0.in‚0" disk.dm-0.out‚23739.0302734375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 network.sent‚ 5.515544e+06 disk.dm-1.out‚ 2255.2265625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚16684.2236328125% disk.nvme0n1.out‚16684.2236328125# memory_percent‚98.4370470046997 proc.memory.rssMB‚ 2964.4375 cpu‚25.81416766990146# disk.nvme1n1.out‚9447.099609375 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚9447.099609375* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17) disk./.usagePercent‚76.39249358420224% disk./.usageGB‚1172.8157577514648 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 256.074218754﬒¯‚(:‡ ’úôÒ‹þ¿ gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚191.787' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚63.929 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý×bˆ2°- _wandbruntime‚2547  _runtime‚2547ï q,ɱà ' lr-AdamW/pg2‚0.00014883571411109238 trainer/global_step‚3799 % _timestamp‚1.7844953912831554e+09 _runtime‚2547.886264502 _step‚309 ' lr-AdamW/pg1‚0.00014883571411109238eŸÎ²È ' lr-AdamW/pg1‚0.00014883571411109238 ' lr-AdamW/pg2‚0.00014883571411109238 trainer/global_step‚3799 % _timestamp‚1.7844953912831554e+09 _runtime‚2547.886264502 _step‚309µ‰ ûÂ2³- _wandbruntime‚2547  _runtime‚2547J³¸à'´ÚO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30090371 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294205148271686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291083637475600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3121527835446 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8599240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30090371 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291083620436240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28787636 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28787448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20188392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8599056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38206589 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68286817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297686514009600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30090373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3223226617344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30090373 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294205252097472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294556286816768 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3130121493504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297686408310272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38196446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220599005445632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294556303856128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3130121493504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3223092979200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 % _timestamp‚1.7844953912837582e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 % _timestamp‚1.7844953912837582e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294205148271686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291083637475600 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3121527835446 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8599240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30090371 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291083620436240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28787636 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28787448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20188392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8599056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38206589 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68286817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297686514009600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30090373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3223226617344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30090373 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294205252097472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294556286816768 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3130121493504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297686408310272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38196446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220599005445632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294556303856128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3130121493504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3223092979200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68304266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3224030748672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294631453522432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294280326117576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 Rí7Ž©EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294631436483072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68304266 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68314425 _runtime‚2548.286142911 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28794865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20193448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220655143173632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28794685 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68314425 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220655049063936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291157928000992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30098000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297762449398784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20193452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223879173922304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3122311330046 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38216423 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8601413 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38206266 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38206266 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291157910961632 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30098000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3130995876352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223878952473088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294280222291678 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30098002 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3130995876352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294631453522432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _step‚311 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3130907209216¤Q|³î&¸èM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3122311330046 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38216423 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8601413 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38206266 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38206266 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚291157910961632 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30098000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3130995876352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223878952473088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294280222291678 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30098002 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3130995876352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294631453522432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _step‚311 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3130907209216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294631436483072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38216423 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3223903409152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3130907209216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68304266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3224030748672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294631453522432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294280326117576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294631436483072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68304266 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68314425 _runtime‚2548.286142911 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28794865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20193448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220655143173632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28794685 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68314425 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220655049063936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291157928000992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30098000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297762449398784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20193452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223879173922304·@øýJ2¹- _wandbruntime‚2548  _runtime‚2548lÝ ¢ºú -loss/FBt1/detection_loss‚0.60477449535283 $loss/total_loss‚9.71498654990982 _step‚312 trainer/global_step‚3799 0loss/FRt1/assignment_loss‚0.8878382369725392 0loss/FRt2/assignment_loss‚2.0324617310474484 1loss/SRqqt1/assignment_loss‚0.619921311989054 2loss/SRqqt2/assignment_loss‚0.7570531544834376 1loss/FBt2/assignment_loss‚0.16682823700487606 .loss/FBt2/detection_loss‚0.626438601605836 # _timestamp‚1.78449539168513e+09 2loss/FBt1/assignment_loss‚0.050887260939093966 .loss/FRt1/detection_loss‚1.529692055064846 /loss/FRt2/detection_loss‚0.6071747191076534 1loss/SRqqt1/detection_loss‚0.6975324399135628 1loss/SRqqt2/detection_loss‚1.1343843064286436 _runtime‚2548.286649502 epoch‚99N`/»ÿ trainer/global_step‚3799 0loss/FRt1/assignment_loss‚0.8878382369725392 0loss/FRt2/assignment_loss‚2.0324617310474484 1loss/SRqqt1/assignment_loss‚0.619921311989054 2loss/SRqqt2/assignment_loss‚0.7570531544834376 1loss/FBt2/assignment_loss‚0.16682823700487606 .loss/FBt2/detection_loss‚0.626438601605836 # _timestamp‚1.78449539168513e+09 2loss/FBt1/assignment_loss‚0.050887260939093966 .loss/FRt1/detection_loss‚1.529692055064846 /loss/FRt2/detection_loss‚0.6071747191076534 1loss/SRqqt1/detection_loss‚0.6975324399135628 1loss/SRqqt2/detection_loss‚1.1343843064286436 _runtime‚2548.286649502 epoch‚9 -loss/FBt1/detection_loss‚0.60477449535283 $loss/total_loss‚9.71498654990982 _step‚312¸H5¼‚(:ª ¡úô҉Ē¿ cpu‚25.812913386896525 network.sent‚ 5.531357e+06% disk./.usageGB‚1172.8157196044922 disk.nvme0n1.in‚0% disk.nvme0n1.out‚16684.9423828125 disk.md0.in‚0 disk.dm-0.in‚0$ memory_percent‚98.43909740447998 network.recv‚706903 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚16684.9423828125 disk.nvme1c1n1.in‚0# disk.nvme1c1n1.out‚ 9447.7734375! disk.nvme1n1.out‚ 9447.7734375 disk.md0.out‚ 63.77734375" disk.dm-0.out‚23739.5732421875 disk.nvme1n1.in‚0 disk.dm-1.out‚2255.529296875 disk.rbd0.out‚0( proc.memory.availableMB‚ 255.73828125 proc.memory.rssMB‚ 2964.4375* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17) disk./.usagePercent‚76.39249109946212 disk.dm-1.in‚0 disk.rbd0.in‚0L?Àzž½‚(:“ ¡úôÒ¶­¿¿ gpu.0.gpu‚63.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.953' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.31766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶€ª4¾‚(:© °úôÒɯâ¾ network.recv‚710835) disk./.usagePercent‚76.39250228079266$ disk.nvme1n1.out‚9448.4970703125 disk.md0.in‚0 disk.md0.out‚64.09375 disk.dm-1.in‚0$ disk.nvme0n1.out‚16687.056640625 disk.nvme1c1n1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0( proc.memory.availableMB‚ 255.70703125* proc.memory.percent‚18.093490600585938 cpu‚25.81148899634609 proc.cpu.threads‚17% disk./.usageGB‚1172.8158912658691 disk.nvme1n1.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚23741.6357421875 disk.dm-1.out‚2255.615234375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚16687.056640625 disk.nvme0n1.in‚0 network.sent‚ 5.58258e+06& disk.nvme1c1n1.out‚9448.4970703125$ memory_percent‚98.43928813934326 proc.memory.rssMB‚ 2964.4375 ÕV{œ¿‚(:‘ °úôÒ⺑¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚114.959' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.31966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LyÌ[2À- _wandbruntime‚2579  _runtime‚2579:°Z3ÈÁ ' lr-AdamW/pg1‚0.00014850030426316927 ' lr-AdamW/pg2‚0.00014850030426316927 trainer/global_step‚3849 $ _timestamp‚1.784495422701102e+09 _runtime‚2579.304284139 _step‚313#¯ÀâÍÂÇ ' lr-AdamW/pg1‚0.00014850030426316927 ' lr-AdamW/pg2‚0.00014850030426316927 trainer/global_step‚3849 $ _timestamp‚1.784495422701102e+09 _runtime‚2579.304284139 _step‚313¹8U2Ã- _wandbruntime‚2579  _runtime‚2579Yß'ÄÙO _runtime‚ 2579.30434666 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38689298 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298313770131968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20441192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚301483347265024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301483241565696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297959016897880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38699441 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30471821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8706401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298313787171328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223405797735936 trainer/global_step‚3849 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20441196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚301483241565696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298313787171328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69171264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚170 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3160766366254 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 % _timestamp‚1.7844954227016776e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30471821 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3169471433728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226669596670464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3169560093696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3169471433728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301483347265024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294798146705840 _step‚314 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚297958913072094 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Cœ_NÅÞO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8706231 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.frbz{ –&eed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38699441 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚223405891845632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3169560093696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29147597 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29147423 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3160853152680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30471823 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38689298 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3263704824832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20441196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚301483241565696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚174 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298313787171328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69171264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚170 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3160766366254 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 % _timestamp‚1.7844954227016776e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30471821 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3169471433728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226669596670464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3169560093696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3169471433728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301483347265024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294798146705840 _step‚314 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚297958913072094 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 2579.30434666 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38689298 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298313770131968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20441192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚301483347265024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301483241565696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297959016897880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38699441 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30471821 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8706401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298313787171328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223405797735936 trainer/global_step‚3849 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29154596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844954231017303e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294872437231232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69188727 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3170257149440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226726537047040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298034090917984 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30479450 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301559282654208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚3849 _step‚315 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30479450 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301559176947712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223462029573632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294872454270592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301559282654208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3161636647392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69178568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38709275 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298388936837632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2579.703688689 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69188727 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20446248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552eô']ï&ÈéM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301559282654208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚3849 _step‚315 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30479450 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301559176947712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚223462029573632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294872454270592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301559282654208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3161636647392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69178568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38709275 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298388936837632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2579.703688689 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69188727 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20446248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30479452 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298388919798272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3170257149440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29154596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844954231017303e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294872437231232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69188727 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3170257149440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226726537047040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298034090917984 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30479450 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360»A;su2É- _wandbruntime‚2579  _runtime‚2579ŒWûÊý 0loss/FRt2/assignment_loss‚2.0808477045566725 1loss/FBt2/assignment_loss‚0.08974226879903924 .loss/FRt1/detection_loss‚1.506711435353436 /loss/FBt1/detection_loss‚0.7430181783183998 _runtime‚2579.704169129 0loss/FRt1/assignment_loss‚0.8544304810072246 2loss/SRqqt1/assignment_loss‚0.6044792544215454 1loss/SRqqt2/detection_loss‚1.0028724243567881 % _timestamp‚1.7844954231026745e+09 %loss/total_loss‚9.708451825218914 epoch‚9 1loss/SRqqt2/assignment_loss‚0.758625694946841 1loss/FBt1/assignment_loss‚0.00649879378220095 .loss/FRt2/detection_loss‚0.579073964074865 .loss/FBt2/detection_loss‚0.816709241808109 1loss/SRqqt1/detection_loss‚0.6654423837937931 trainer/global_step‚3849 _step‚316òæøÖË‚ _runtime‚2579.704169129 0loss/FRt1/assignment_loss‚0.8544304810072246 2loss/SRqqt1/assignment_loss‚0.6044792544215454 1loss/SRqqt2/detection_loss‚1.0028724243567881 % _timestamp‚1.7844954231026745e+09 %loss/total_loss‚9.708451825218914 epoch‚9 1loss/SRqqt2/assignment_loss‚0.758625694946841 1loss/FBt1/assignment_loss‚0.00649879378220095 .loss/FRt2/detection_loss‚0.579073964074865 .loss/FBt2/detection_loss‚0.816709241808109 1loss/SRqqt1/detection_loss‚0.6654423837937931 trainer/global_step‚3849 _step‚316 0loss/FRt2/assignment_loss‚2.0808477045566725 1loss/FBt2/assignment_loss‚0.08974226879903924 .loss/FRt1/detection_loss‚1.506711435353436 /loss/FBt1/detection_loss‚0.7430181783183998¼\8Ñ[@Ì‚(:µ ¿úôÒî¸Ó¿) disk./.usagePercent‚76.39250054147456 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 64.62109375 disk.dm-0.in‚0# disk.dm-0.out‚23742.54638671875" disk.dm-1.out‚2255.82275390625$ memory_percent‚98.44088554382324' disk.nvme1c1n1.out‚9449.39111328125 disk.rbd0.out‚0 cpu‚25.81024757489145 network.sent‚ 5.598393e+06 network.recv‚714262% disk./.usageGB‚1172.8158645629883 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚16688.38818359375& disk.nvme0n1.out‚16688.38818359375% disk.nvme1n1.out‚9449.39111328125 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0' proc.memory.availableMB‚ 255.4453125 proc.memory.rssMB‚ 2964.4375* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17À®b`žÍ‚(:“ ¿úôÒÚçÞ¿ gpu.0.gpu‚54.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.722' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.24066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_«g>΂(:³ ÎúôÒ›•ö¾' disk.nvme1c1n1.out‚9450.49267578125 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.093490600585938$ disk./.usageGB‚1172.815746307373 disk.dm-0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚2255.93212890625( proc.memory.availableMB‚ 255.80078125 network.sent‚ 5.649938e+06( disk.nvme0c0n1.out‚16691.40771484375 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚23745.30029296875 proc.memory.rssMB‚ 2964.4375 cpu‚25.808958056062668( disk./.usagePercent‚76.3924928387802 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚16691.40771484375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚9450.49267578125 disk.md0.out‚ 65.203125$ memory_percent‚98.43871593475342 proc.cpu.threads‚17 network.recv‚718272ä >žÏ‚(:“ ÎúôÒ’½£¿ gpu.0.gpu‚66.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚223.945' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.64833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Éh%ˆ@Ђ(:µ ÝúôÒ¤àÖ¾( disk.nvme0c0n1.out‚16692.83251953125 proc.memory.rssMB‚ 2964.4375 network.sent‚ 5.652902e+06% disk.nvme1n1.out‚9451.81982421875 disk.md0.in‚0 disk.md0.out‚ 66.01171875& disk.nvme0n1.out‚16692.83251953125' disk.nvme1c1n1.out‚9451.81982421875$ memory_percent‚98.44045639038086& proc.memory.availableMB‚ 255.515625" disk.dm-1.out‚2256.19775390625 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.30540443697458% disk./.usageGB‚1171.4787216186523# disk.dm-0.out‚23746.11669921875 disk.dm-1.in‚0* proc.memory.percent‚18.093490600585938 cpu‚25.807673267339098 network.recv‚721787 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0ø©å Ñ‚(:„ ÝúôÒýùš¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚114.75' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.25 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_‹2Ò- _wandbruntime‚2610  _runtime‚2610QŒÙÇÓÁ _runtime‚2610.361594152 _step‚317 & lr-AdamW/pg1‚0.0001481605092276829 & lr-AdamW/pg2‚0.0001481605092276829 trainer/global_step‚3899 % _timestamp‚1.7844954537596867e+09Û™<‹ÌÔÆ & lr-AdamW/pg1‚0.0001481605092276829 & lr-AdamW/pg2‚0.0001481605092276829 trainer/global_step‚3899 % _timestamp‚1.7844954537596867e+09 _runtime‚2610.361594152 _step‚317½¤Øá2Õ- _wandbruntime‚2610  _runtime‚2610úèÆà'ÖÚO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30853273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20693992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226212684135936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302071253447168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70045566 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3208910033920 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚166 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29506520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226212778245632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3304171385344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30853273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305280180520448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚305280074821120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3304050330112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298512690014800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3208821373952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302071270486528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3208910033920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39182150 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301712781698288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302071253447168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚318 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39192293 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3200004897062 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302071270486528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39192293 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8812528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30853271 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229516734466048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70035421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301712677872502 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305280074821120 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30853271 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20693996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚305280180520448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚305280074821120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3304050330112 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298512690014800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3208821373952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302071270486528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3208910033920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39182150 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301712781698288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302071253447168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 _step‚318 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39192293 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3200004897062 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302071270486528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39192293 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8812528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30853271 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229516734466048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70035421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301712677872502 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305280074821120 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30853271 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20693996 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301787855718392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20699048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3200788391662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298586963500832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3200875178200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305356115909632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3304979932160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70063029 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30860900 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70063029 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305356010203136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70052870 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29513831 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302146403113472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚163 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20699052 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302146420152832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚319 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226268915973632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301787751892494 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3209607089664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3209607089664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29513664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305356115909632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3304856786944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298586980540192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70052870 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229573678650880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39191970 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302146403113472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3209695756800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30860902 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.ƒ1ÌèÖallocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚167 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229573895905792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30860900 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784495454161835e+09\+’Îî&ÚèM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3304856786944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298586980540192 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70052870 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229573678650880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39191970 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302146403113472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3209695756800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30860902 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚167 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229573895905792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30860900 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784495454161835e+09 _runtime‚2610.763807599 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39191970 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301787855718392 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20699048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3200788391662 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298586963500832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3200875178200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305356115909632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3304979932160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70063029 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30860900 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70063029 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305356010203136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70052870 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29513831 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302146403113472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚163 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20699052 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302146420152832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚319 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚226268915973632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301787751892494 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3209607089664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3209607089664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29513664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305356115909632¿R[422Û- _wandbruntime‚2610  _runtime‚2610Ò ïfÜ€ 0loss/FRt1/assignment_loss‚0.9188332481167084 0loss/FRt2/assignment_loss‚2.0955645926609763 0loss/FBt2/assignment_loss‚0.2717059374105882 /loss/FRt1/detection_loss‚1.4837693265382794 /loss/FBt1/detection_loss‚0.5720293357451336 %loss/total_loss‚9.725541493403993 % _timestamp‚1.7844954541627533e+09 _step‚320 /loss/FRt2/detection_loss‚0.5885403202196549 1loss/SRqqt1/detection_loss‚0.6512858679053315 1loss/SRqqt2/detection_loss‚1.0282355085084285 /loss/FBt2/detection_loss‚0.8659588112946479 _runtime‚2610.764323319 epoch‚9 trainer/global_step‚3899 2loss/SRqqt1/assignment_loss‚0.5835599130218228 2loss/SRqqt2/assignment_loss‚0.6097361814817068 1loss/FBt1/assignment_loss‚0.05632245050071482õ„ŸS Ý… 1loss/FBt1/assignment_loss‚0.05632245050071482 0loss/FRt1/assignment_loss‚0.9188332481167084 0loss/FRt2/assignment_loss‚2.0955645926609763 0loss/FBt2/assignment_loss‚0.2717059374105882 /loss/FRt1/detection_loss‚1.4837693265382794 /loss/FBt1/detection_loss‚0.5720293357451336 %loss/total_loss‚9.725541493403993 % _timestamp‚1.7844954541627533e+09 _step‚320 /loss/FRt2/detection_loss‚0.5885403202196549 1loss/SRqqt1/detection_loss‚0.6512858679053315 1loss/SRqqt2/detection_loss‚1.0282355085084285 /loss/FBt2/detection_loss‚0.8659588112946479 _runtime‚2610.764323319 epoch‚9 trainer/global_step‚3899 2loss/SRqqt1/assignment_loss‚0.5835599130218228 2loss/SRqqt2/assignment_loss‚0.6097361814817068À î-b?Þ‚(:´ ìúôÒù–ñ¾) disk./.usagePercent‚76.30545363482891# disk.dm-0.out‚23753.19384765625 disk.md0.out‚ 66.3671875* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.dm-1.out‚2256.28369140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚16700.01513671875 disk.md0.in‚0 cpu‚25.80639387918596$ disk./.usageGB‚1171.479476928711 disk.nvme0n1.in‚0& disk.nvme0n1.out‚16700.01513671875% disk.nvme1n1.out‚9452.58935546875( proc.memory.availableMB‚ 256.99609375 proc.memory.rssMB‚ 2964.4375 network.recv‚724942 disk.dm-0.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9452.58935546875 disk.nvme1n1.in‚0$ memory_percent‚98.43142032623291 network.sent‚ 5.658732e+06¥ÚÈžß‚(:“ ìúôÒ°‡Ï¿ gpu.0.gpu‚55.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚218.446' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚72.81533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÛ:à‚(:¯ ûúôÒ©öÔ¾) disk./.usagePercent‚76.30548693034649 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 2256.546875 disk.rbd0.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 257.24609375 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚9453.2626953125 proc.memory.rssMB‚ 2964.4375 cpu‚25.80504343896942 network.sent‚ 5.720962e+06% disk./.usageGB‚1171.4799880981445 disk.md0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚16701.11669921875& disk.nvme0n1.out‚16701.11669921875* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17 disk.md0.out‚ 66.62890625# disk.dm-0.out‚23754.11962890625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚9453.2626953125$ memory_percent‚98.42989444732666 network.recv‚7290582†žžá‚(:“ ûúôÒµ–¿ gpu.0.gpu‚64.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.839' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.27966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Œ2â- _wandbruntime‚2641  _runtime‚2641ur¬Éãà trainer/global_step‚3949 % _timestamp‚1.7844954852276492e+09 _runtime‚2641.829477622 _step‚321 ' lr-AdamW/pg1‚0.00014781635082903346 ' lr-AdamW/pg2‚0.00014781635082903346¸ð²/ÎäÈ ' lr-AdamW/pg1‚0.00014781635082903346 ' lr-AdamW/pg2‚0.00014781635082903346 trainer/global_step‚3949 % _timestamp‚1.7844954852276492e+09 _runtime‚2641.829477622 _step‚321ÁÒÂd–2å- _wandbruntime‚2641  _runtime‚2641™¡Oxà'æÚO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302227199245040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302227216284400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305828736762368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚229019664645632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309077013775872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3344523287040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 trainer/global_step‚3949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3248259974144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚172 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305466442672910 _runtime‚2641.830415053 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚309076908076544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232364093822976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70919868 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39675002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229019570535936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3239243427870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309076908076544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305828753801728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305466546498696 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31234723 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8918556 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31234721 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3248259974144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309077013775872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3344644342272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39685145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3248171314176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29865348 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8918724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232364308987904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8918724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232364308987904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309077013775872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3344523287040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 trainer/global_step‚3949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3248259974144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚172 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305466442672910 _runtime‚2641.830415053 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚309076908076544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232364093822976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70919868 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39675002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229019570535936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3239243427870 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309076908076544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305828753801728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚168 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305466546498696 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31234723 SGDeviceStatsMonitor.on_train_batch_start/inacŽFöDå tive_split.small_pool.freed‚8918556 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31234721 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3248259974144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309077013775872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3344644342272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39685145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3248171314176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29865348 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29872465 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70927172 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20951848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302301489770432 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20951852 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229075708263936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3345334700032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31242352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232421258121728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215157760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39694979 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39684822 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70937331 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31242350 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3248957029888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3249045697024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302301506809792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844954856299236e+09 _step‚323 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8920793 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3249045697024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3240026922470 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _runtime‚2642.231895729 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31242350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39684822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309152949165056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚3949 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8920617 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29872645 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232421042963968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305541620518800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3240113709008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309152843458560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3345455748096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229075802373632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130ï Dï&êéM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215157760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39694979 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39684822 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70937331 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31242350 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3248957029888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3249045697024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302301506809792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 % _timestamp‚1.7844954856299236e+09 _step‚323 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8920793 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3249045697024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3240026922470 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _runtime‚2642.231895729 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31242350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39684822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309152949165056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚3949 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8920617 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 121048064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29872645 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232421042963968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305541620518800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3240113709008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309152843458560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3345455748096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229075802373632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305903903468032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305903886428672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70927172 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70937331 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31242352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29872465 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70927172 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20951848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302301489770432 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20951852 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229075708263936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3345334700032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31242352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232421258121728ÃT]”2ë- _wandbruntime‚2642  _runtime‚2642Òfìû 1loss/SRqqt2/detection_loss‚1.1144432793090107 /loss/FBt1/detection_loss‚0.5518638357999875 2loss/FBt1/assignment_loss‚0.011102750197835092 /loss/FRt1/detection_loss‚1.4518118146454033 $ _timestamp‚1.784495485630861e+09 _step‚324 /loss/FBt2/assignment_loss‚0.176005833397361 /loss/FBt2/detection_loss‚0.6097581962054525 #loss/total_loss‚9.6654554912657 trainer/global_step‚3949 _runtime‚ 2642.23235474 0loss/FRt2/assignment_loss‚2.2010652307245033 2loss/SRqqt1/assignment_loss‚0.6538115431129216 2loss/SRqqt2/assignment_loss‚0.7510610324168115 .loss/FRt2/detection_loss‚0.573616405035692 epoch‚9 0loss/FRt1/assignment_loss‚0.8945535255733289 1loss/SRqqt1/detection_loss‚0.6763620448473922Žr{í€ /loss/FBt2/assignment_loss‚0.176005833397361 /loss/FBt2/detection_loss‚0.6097581962054525 #loss/total_loss‚9.6654554912657 trainer/global_step‚3949 _runtime‚ 2642.23235474 0loss/FRt2/assignment_loss‚2.2010652307245033 2loss/SRqqt1/assignment_loss‚0.6538115431129216 2loss/SRqqt2/assignment_loss‚0.7510610324168115 .loss/FRt2/detection_loss‚0.573616405035692 epoch‚9 0loss/FRt1/assignment_loss‚0.8945535255733289 1loss/SRqqt1/detection_loss‚0.6763620448473922 1loss/SRqqt2/detection_loss‚1.1144432793090107 /loss/FBt1/detection_loss‚0.5518638357999875 2loss/FBt1/assignment_loss‚0.011102750197835092 /loss/FRt1/detection_loss‚1.4518118146454033 $ _timestamp‚1.784495485630861e+09 _step‚324ÄJBÚ£8î‚(:­ ŠûôÒÁßî¾ disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 network.sent‚ 5.724106e+06) disk./.usagePercent‚76.30543698707012 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚16705.6240234375% disk.nvme0n1.out‚16705.6240234375 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0! disk.dm-0.out‚23758.724609375 disk.dm-1.in‚0 disk.dm-1.out‚ 2256.66015625$ memory_percent‚98.43013286590576& disk.nvme1c1n1.out‚9454.5751953125 network.recv‚732282% disk./.usageGB‚1171.4792213439941 disk.md0.out‚ 67.1484375* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17$ disk.nvme1n1.out‚9454.5751953125( proc.memory.availableMB‚ 257.20703125 proc.memory.rssMB‚ 2964.4375 cpu‚25.803980583210784¶O¡s‘ï‚(:† ŠûôÒ—ÃÅ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚136.827' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚45.609 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷›«Ç=ð‚(:² ™ûôÒµî¾ disk.dm-1.in‚0! disk.dm-1.out‚2256.7451171875( disk.nvme0c0n1.out‚16706.66748046875 proc.cpu.threads‚17 disk.dm-0.in‚0! disk.dm-0.out‚23759.337890625% disk./.usageGB‚1171.4792251586914 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9455.32470703125 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 network.recv‚736471' disk.nvme1c1n1.out‚9455.32470703125 disk.nvme0n1.in‚0& disk.nvme0n1.out‚16706.66748046875 proc.memory.rssMB‚ 2964.4375 cpu‚25.802747492956417 network.sent‚ 5.788177e+06) disk./.usagePercent‚76.30543723554413 disk.nvme1c1n1.in‚0 disk.md0.out‚ 67.6640625 disk.rbd0.out‚0# memory_percent‚98.4302282333374( proc.memory.availableMB‚ 257.19140625* proc.memory.percent‚18.093490600585938UJ[“ñ‚(:ˆ ™ûôÒö³›¿ gpu.0.gpu‚54.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚223.311' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚74.437 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"r2ò- _wandbruntime‚2672  _runtime‚2672¯¸{³Éóà ' lr-AdamW/pg1‚0.00014746785117187178 ' lr-AdamW/pg2‚0.00014746785117187178 trainer/global_step‚3999 % _timestamp‚1.7844955162972126e+09 _runtime‚2672.899069921 _step‚325aÕ…HÎôÈ trainer/global_step‚3999 % _timestamp‚1.7844955162972126e+09 _runtime‚2672.899069921 _step‚325 ' lr-AdamW/pg1‚0.00014746785117187178 ' lr-AdamW/pg2‚0.00014746785117187178ÅÿÓHh2õ- _wandbruntime‚2672  _runtime‚2672Á?ÀÄÓöÙO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312873847031296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30224635 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235211657586176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3278568745104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3278481958678 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40167854 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3287521254400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71784025 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31616173 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9025039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312873741331968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231826456935936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚194 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30224441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚309586237116928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31616173 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _runtime‚2672.903235827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3999 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40167854 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 $ _timestamp‚1.784495516298126e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309586237116928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚190 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309220311299104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305941742554000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3287521254400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3385106540544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812‹ÖDä'÷ÞO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40167854 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 $ _timestamp‚1.784495516298126e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309586237116928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚190 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309220311299104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305941742554000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3287521254400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3385106540544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312873847031296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30224635 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235211657586176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3278568745104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3278481958678 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40167854 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3287521254400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71784025 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31616173 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9025039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312873741331968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231826456935936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚194 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30224441 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚309586237116928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31616173 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _runtime‚2672.903235827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3999Æ­,d2ø- _wandbruntime‚2673  _runtime‚2673ç3\3ê&ùäM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40177674 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309661369743872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306016033079392 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231882688773632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31623800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3288306970112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3999 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3279352239816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31623802 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚2673.300796885 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40187831 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231882594663936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71801474 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31623802 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40187831 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3288306970112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3385909713408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3279265453278 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71801474 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312949782420480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71811633 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21204652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235268372843520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3288395637248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31623800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21204648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30231635 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9027161 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306016016040032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71811633 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3288395637248ª ¿¼ÑúéM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40177674 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312949676713984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30231813 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3385778179584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309295281493310 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312949782420480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚309661369743872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235268598487040 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71811633 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3288395637248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40177674 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309661369743872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306016033079392 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231882688773632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31623800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3288306970112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3999 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3279352239816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31623802 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚2673.300796885 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40187831 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231882594663936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71801474 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31623802 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40187831 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3288306970112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3385909713408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3279265453278 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71801474 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312949782420480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71811633 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21204652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235268372843520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segmend› t.large_pool.current‚682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3288395637248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31623800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21204648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30231635 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9027161 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306016016040032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 P&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 (particle/accuracy‚0.8474353551864624 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9871954917907715 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8062639832496643 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21045464277267456 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.642042875289917 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9399287700653076 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.292460173368454 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18201936781406403 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7249007225036621 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38210397958755493 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6473338007926941 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.29941526055336 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.09808306396007538 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6827751994132996 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.444821298122406 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9856928586959839 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6674786806106567 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2401115596294403 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7886999249458313 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7213890552520752 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7710376977920532 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2442987710237503 *jet/accuracy_1_of_1‚0.3373390734195709 +jet/accuracy_2_of_2‚0.14490903913974762 _step‚329 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6860476732254028 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.036731377243995667 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881234169006348 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.989258885383606 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 /particle/accuracy_3_of_4‚0.9942177534103394 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_3‚0.0705341249704361 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8931394219398499 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897379875183105 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39916175603866577 'particle/f_score‚0.6334581971168518 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32679781317710876 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.674003005027771 epoch‚9 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4726555347442627 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.647712767124176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1798735111951828 particle/accuracy_1_of_1‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2388879954814911 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8006425499916077 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6450151801109314 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +particle/sensitivity‚0.6138933897018433 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416865348816 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6908438801765442 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5541442036628723 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8854279518127441 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7339354753494263 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03225688263773918 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6256348490715027 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8059369325637817 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0760076716542244 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7844955301219568e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982777833938599 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3819444477558136 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6519206166267395 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.039388809353113174 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910464286804199 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2324703335762024 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5866380333900452 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2028549760580063 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43200311064720154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7977476716041565 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5478249192237854 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8357268571853638 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06160970404744148 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13463439047336578 5validation_average_jet_accuracy‚0.375305712223053 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 trainer/global_step‚3999 )loss/val_total_loss‚9.232802772030347 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5769749879837036 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6989912986755371 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.200332373380661 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7834278345108032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10662313550710678 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7267760038375854 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25047388672828674 (jet/accuracy_2_of_3‚0.53285151720047 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19298392534255981 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03866519406437874 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9775324463844299 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6171684861183167 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7513983845710754 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6871408820152283 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.036352984607219696 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7298874855041504 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.457956999540329 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27873480319976807 particle/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19144773483276367 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5513358116149902 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.463256299495697 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7731087803840637 *jet/accuracy_1_of_2‚0.7804304957389832 +particle/specificity‚0.9113068580627441 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8434805274009705 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9723278880119324 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19725608825683594 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1980879008769989 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7902116179466248 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9399340748786926 ,jet/accuracy_4_of_4‚0.011098379269242287 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974949955940247 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2401115596294403 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046437907963991165 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972758293151855 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19661594927310944 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9917368292808533 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6474054455757141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7328281998634338 *jet/accuracy_1_of_3‚0.9171183705329895 particle/accuracy_1_of_4‚1„‘Ö¶,‚Ü{ 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5933374762535095 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6129336357116699 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.032195065170526505 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11760107427835464 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.996021568775177 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974949955940247 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16499999165534973 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5429166555404663 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8899842500686646 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38197803497314453 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21208594739437103 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12309122085571289 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963982105255127 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6869425177574158 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8434805274009705 jet/accuracy_4_of_5‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03777375444769859 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_4‚0.9732686877250671 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49748602509498596 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25075316429138184 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6380080580711365 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6157525181770325 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03225688263773918 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9641884565353394 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9699670076370239 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7048876881599426 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8463734984397888 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 _runtime‚2686.724183596 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6474054455757141 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18363654613494873 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6425100564956665 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03620004653930664 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 (particle/accuracy‚0.8474353551864624 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9027777910232544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9871954917907715 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8062639832496643 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21045464277267456 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.642042875289917 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9399287700653076 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.292460173368454 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18201936781406403 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7249007225036621 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38210397958755493 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6473338007926941 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.29941526055336 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.09808306396007538 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6827751994132996 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.444821298122406 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9856928586959839 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6674786806106567 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2401115596294403 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7886999249458313 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7213890552520752 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7710376977920532 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2442987710237503 *jet/accuracy_1_of_1‚0.3373390734195709 +jet/accuracy_2_of_2‚0.14490903913974762 _step‚329 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6860476732254028 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.036731377243995667 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9881234169006348 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.989258885383606 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 /particle/accuracy_3_of_4‚0.9942177534103394 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_3‚0.0705341249704361 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8931394219398499 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897379875183105 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39916175603866577 'particle/f_score‚0.6334581971168518 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32679781317710876 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.674003005027771 epoch‚9 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4726555347442627 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.647712767124176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1798735111951828 particle/accuracy_1_of_1‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2388879954814911 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8006425499916077 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6450151801109314 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +particle/sensitivity‚0.6138933897018433 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416865348816 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6908438801765442 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5541442036628723 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8854279518127441 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7339354753494263 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03225688263773918 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6256348490715027 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8059369325637817 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0760076716542244 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7844955301219568e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982777833938599 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3819444477558136 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6519206166267395 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.039388809353113174 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910464286804199 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2324703335762024 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5866380333900452 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2028549760580063 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43200311064720154 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7977476716041565 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5478249192237854 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8357268571853638 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06160970404744148 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13463439047336578 5validation_average_jet_accuracy‚0.375305712223053 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 trainer/global_step‚3999 )loss/val_total_loss‚9.232802772030347 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5769749879837036 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6989912986755371 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.200332373380661 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7834278345108032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10662313550710678 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7267760038375854 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.^cQS9814814329147339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25047388672828674 (jet/accuracy_2_of_3‚0.53285151720047 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19298392534255981 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03866519406437874 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9775324463844299 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6171684861183167 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7513983845710754 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6871408820152283 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.036352984607219696 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7298874855041504 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.457956999540329 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27873480319976807 particle/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19144773483276367 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5513358116149902 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.463256299495697 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7731087803840637 *jet/accuracy_1_of_2‚0.7804304957389832 +particle/specificity‚0.9113068580627441 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8434805274009705 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9723278880119324 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19725608825683594 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1980879008769989 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7902116179466248 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9399340748786926 ,jet/accuracy_4_of_4‚0.011098379269242287 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974949955940247 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2401115596294403 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046437907963991165 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972758293151855 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19661594927310944 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9917368292808533 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6474054455757141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7328281998634338 *jet/accuracy_1_of_3‚0.9171183705329895 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2512470781803131 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7366629242897034 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22428005933761597ÉÍ¢àžƒ‚(:“ ·ûôÒü‚¿ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚157.886' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚52.62866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü¾¯.„‚(:£ ·ûôÒÐܪÁ disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚9520.6767578125* proc.memory.percent‚18.093490600585938 proc.cpu.threads‚17 network.recv‚743862) disk./.usagePercent‚76.30542207862942 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚9520.6767578125 disk.md0.in‚0 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚16710.087890625! disk.dm-1.out‚2320.8505859375 disk.nvme1c1n1.in‚0$ memory_percent‚98.43082427978516 proc.memory.rssMB‚ 2964.4375 cpu‚25.815288022795393% disk./.usageGB‚1171.4789924621582 disk.md0.out‚68.5 disk.dm-0.out‚23762.20703125 disk.rbd0.in‚0$ disk.nvme0n1.out‚16710.087890625% proc.memory.availableMB‚ 257.09375 network.sent‚ 5.854332e+06 disk.dm-1.in‚0 disk.rbd0.out‚0CO&{=…‚(:² ÆûôÒ´“í¾$ memory_percent‚98.43602180480957' proc.memory.availableMB‚ 256.2421875 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚16712.27685546875 proc.memory.rssMB‚ 2964.4375* proc.memory.percent‚18.093490600585938 disk.md0.out‚ 68.90234375" disk.dm-0.out‚23764.3212890625 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 network.sent‚ 5.912053e+06( disk./.usagePercent‚76.3054322660639$ disk./.usageGB‚1171.479148864746 disk.dm-0.in‚0 cpu‚25.814153849212133 proc.cpu.threads‚17 network.recv‚747851& disk.nvme0n1.out‚16712.27685546875' disk.nvme1c1n1.out‚9557.56396484375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9557.56396484375! disk.dm-1.out‚2356.9365234375[=A ž†‚(:“ ÆûôÒ˜Ö ¿ gpu.0.gpu‚91.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.042' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.34733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"›¨ì2‡- _wandbruntime‚2718  _runtime‚2718[‘ïQɈà % _timestamp‚1.7844955616974528e+09 _runtime‚2718.299284777 _step‚330 ' lr-AdamW/pg1‚0.00014711503263967963 ' lr-AdamW/pg2‚0.00014711503263967963 trainer/global_step‚4049†%Î‰È ' lr-AdamW/pg1‚0.00014711503263967963 ' lr-AdamW/pg2‚0.00014711503263967963 trainer/global_step‚4049 % _timestamp‚1.7844955616974528e+09 _runtime‚2718.299284777 _step‚330Ê oä’2Š- _wandbruntime‚2718  _runtime‚2718îr[Tà'‹ÚO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21560647 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40753356 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317682491260416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3333135748052 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314340207728128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9179282 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310629718820432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72862634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314340190688768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40753356 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3342300571648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313962958394270 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3441203254272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441078004736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317682491260416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30739929 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9179454 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72872779 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21560651 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238877786545664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235436708540928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313962854568484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32119423 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238878005904896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844955616983874e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3333222534478 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40743213 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314340207728128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314340190688768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚2718.300177828 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317682596959744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32119423 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32119421 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32119421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3342389231616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 _step‚331 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30740105 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310629735859792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235436802650624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760=M—)å'ŒßO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3441203254272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441078004736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317682491260416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30739929 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9179454 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72872779 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21560651 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238877786545664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235436708540928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313962854568484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32119423 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238878005904896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844955616983874e+09 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3333222534478 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40743213 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314340207728128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314340190688768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚2718.300177828 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317682596959744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32119423 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32119421 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32119421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3342389231616 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 _step‚331 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30740105 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310629735859792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235436802650624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21560647 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40753356 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317682491260416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3333135748052 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314340207728128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9179282 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310629718820432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72862634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314340190688768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40753356 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3342300571648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313962958394270 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374Ë[®Žˆ2- _wandbruntime‚2718  _runtime‚2718«8¶ö¹ŽäM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9181349 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238934737570816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72890242 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21565707 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72880083 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127052 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310704009345824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314415357394432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40753033 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3441891301888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238934950631424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235492940378624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚4049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314415357394432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9181510 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3333919242652 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72880083 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40753033 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚161 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚165 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310704026385184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40763190 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJø»Æ 1DeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72890242 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3343174954496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21565703 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314038032414374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314037928588476 _runtime‚2718.700773257 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3343086287360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 _step‚332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30747052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314415340355072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3334006029190 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317758532348928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317758532348928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 % _timestamp‚1.7844955620987792e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32127050 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812?ùläï&éM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72890242 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3343174954496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21565703 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314038032414374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314037928588476 _runtime‚2718.700773257 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3343086287360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 _step‚332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30747052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314415340355072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3334006029190 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317758532348928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317758532348928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 % _timestamp‚1.7844955620987792e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32127050 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9181349 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238934737570816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72890242 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21565707 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72880083 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127052 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310704009345824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314415357394432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40753033 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3441891301888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238934950631424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235492940378624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚4049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314415357394432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9181510 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3333919242652 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72880083 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40753033 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚161 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚165 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310704026385184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40763190 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488̤†}n2- _wandbruntime‚2718  _runtime‚2718óN… ‘ÿ 1loss/FBt2/assignment_loss‚0.09571997509110047 -loss/FRt2/detection_loss‚0.55304035546735 1loss/SRqqt1/detection_loss‚0.6481693409102026 1loss/SRqqt2/detection_loss‚1.1633098551369687 2loss/FBt1/assignment_loss‚0.027464163666448504 /loss/FBt1/detection_loss‚0.7711604848589878 %loss/total_loss‚9.824584692681073 epoch‚10 _runtime‚2718.701309107 _step‚333 0loss/FRt1/assignment_loss‚0.9318288301895998 1loss/SRqqt1/assignment_loss‚0.653639890408711 /loss/FBt2/detection_loss‚0.6493381623215243 % _timestamp‚1.7844955620997488e+09 /loss/FRt2/assignment_loss‚2.173301274759498 2loss/SRqqt2/assignment_loss‚0.6431795561411289 /loss/FRt1/detection_loss‚1.5144328037295527 trainer/global_step‚4049N–g ’„ 1loss/SRqqt1/assignment_loss‚0.653639890408711 /loss/FBt2/detection_loss‚0.6493381623215243 % _timestamp‚1.7844955620997488e+09 /loss/FRt2/assignment_loss‚2.173301274759498 2loss/SRqqt2/assignment_loss‚0.6431795561411289 /loss/FRt1/detection_loss‚1.5144328037295527 trainer/global_step‚4049 1loss/FBt2/assignment_loss‚0.09571997509110047 -loss/FRt2/detection_loss‚0.55304035546735 1loss/SRqqt1/detection_loss‚0.6481693409102026 1loss/SRqqt2/detection_loss‚1.1633098551369687 2loss/FBt1/assignment_loss‚0.027464163666448504 /loss/FBt1/detection_loss‚0.7711604848589878 %loss/total_loss‚9.824584692681073 epoch‚10 _runtime‚2718.701309107 _step‚333 0loss/FRt1/assignment_loss‚0.9318288301895998ÍêÀw9“‚(:® ÕûôÒ»ìî¾( proc.memory.availableMB‚ 256.19140625 cpu‚25.812891788794705 disk.md0.out‚ 69.1796875" disk.dm-1.out‚2384.90478515625 disk.nvme0c0n1.in‚0 proc.memory.rssMB‚ 2964.4375 proc.cpu.threads‚17) disk./.usagePercent‚76.30542356947349 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9585.95068359375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9585.95068359375" disk.dm-0.out‚23770.2861328125 disk.rbd0.in‚0 network.sent‚ 5.915017e+06% disk./.usageGB‚1171.4790153503418# disk.nvme0n1.out‚16718.44140625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚16718.44140625* proc.memory.percent‚18.093490600585938 network.recv‚751459 disk.dm-0.in‚0# memory_percent‚98.4363317489624Ag$ ž”‚(:“ ÕûôÒ‡ÁÏ¿ gpu.0.gpu‚95.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.034' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.34466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁÐÓy8•‚(:­ äûôÒ­½å¾ disk.md0.out‚69.59375 disk.dm-1.in‚0" disk.dm-1.out‚2385.01416015625 disk.rbd0.in‚0 proc.memory.rssMB‚ 2964.4375 cpu‚25.811645109002058 proc.cpu.threads‚17$ disk./.usageGB‚1171.436122894287 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚16720.5244140625% disk.nvme0n1.out‚16720.5244140625& disk.nvme1c1n1.out‚9587.0009765625 disk.nvme1n1.in‚0 network.sent‚ 5.979176e+06 network.recv‚755749 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚9587.0009765625$ memory_percent‚98.43716621398926' proc.memory.availableMB‚ 256.0546875( disk./.usagePercent‚76.3026297276852# disk.dm-0.out‚23772.41259765625* proc.memory.percent‚18.093490600585938³£¨ž–‚(:“ äûôÒ§„˜¿ gpu.0.gpu‚86.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚225.496' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.16533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï?#–2—- _wandbruntime‚2749  _runtime‚2749‰V¦˜É˜Ã _step‚334 ' lr-AdamW/pg1‚0.00014675791789333203 ' lr-AdamW/pg2‚0.00014675791789333203 trainer/global_step‚4099 % _timestamp‚1.7844955928133495e+09 _runtime‚2749.415187022Èæ¢Î™È ' lr-AdamW/pg1‚0.00014675791789333203 ' lr-AdamW/pg2‚0.00014675791789333203 trainer/global_step‚4099 % _timestamp‚1.7844955928133495e+09 _runtime‚2749.415187022 _step‚334ÎmËoè2š- _wandbruntime‚2749  _runtime‚2749Ú@û¹à'›ÚO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31100958 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321479430215168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9287511 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3481826582528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31101140 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9287689 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73736936 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32500873 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3381650511872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318097674003968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41246208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3381650511872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3481705527296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321479324515840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317716619368892 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚2749.415926083 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318097691043328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241725300468224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32500871 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚4099 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321479324515840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32500871 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41246208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317716723194678 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41236065 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318097674003968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314344245090032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21813447 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3381739171840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241725515633152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41246208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3381650511872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3481705527296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321479324515840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317716619368892 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚2749.415926083 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318097691043328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241725300468224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32500871 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚182 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚4099 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321479324515840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32500871 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41246208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317716723194678 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41236065 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318097674003968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314344245090032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21813447 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3381739171840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241725515633152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200Ï– ò2- _wandbruntime‚2749  _runtime‚2749TSSé&žãM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3482629100032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3482518537728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318172823670272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238299826778624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317791693388884 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241782251206656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318172840709632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3373157773460 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321555365604352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 _step‚336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32508500 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3373244559998 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32508502 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110562304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73754385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204672000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41256042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31108358 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31108169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚2749.816953332 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32508500 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321555259897856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73764544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314418535615424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41256042 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73764544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21818507 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9289851 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32508502 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚318172823670272“õÇî&ŸèM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41256042 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73764544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21818507 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9289851 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32508502 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚318172823670272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3482629100032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3482518537728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318172823670272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238299826778624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317791693388884 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241782251206656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318172840709632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3373157773460 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321555365604352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 _step‚336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32508500 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3373244559998 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32508502 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110562304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73754385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 204672000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41256042 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31108358 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31108169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚2749.816953332 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32508500 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321555259897856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73764544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314418535615424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960ÐÒœøþ2 - _wandbruntime‚2749  _runtime‚27492 ¡€ /loss/FRt2/assignment_loss‚2.085968956917147 2loss/SRqqt1/assignment_loss‚0.6082559236453349 1loss/SRqqt1/detection_loss‚0.6215308151427027 _step‚337 %loss/total_loss‚9.547821192384504 epoch‚10 0loss/FRt1/assignment_loss‚0.8259838319135074 2loss/SRqqt2/assignment_loss‚0.7091113821058932 2loss/FBt1/assignment_loss‚0.004187971076421542 /loss/FRt2/detection_loss‚0.5932862959735725 1loss/SRqqt2/detection_loss‚1.1149556993517236 /loss/FBt1/detection_loss‚0.8094932254707384 _runtime‚2749.817455013 0loss/FBt2/assignment_loss‚0.0866751856588271 /loss/FRt1/detection_loss‚1.5587914224124135 .loss/FBt2/detection_loss‚0.529580482716223 trainer/global_step‚4099 % _timestamp‚1.7844955932159123e+099ÏÜ# ¢… /loss/FRt2/assignment_loss‚2.085968956917147 2loss/SRqqt1/assignment_loss‚0.6082559236453349 1loss/SRqqt1/detection_loss‚0.6215308151427027 _step‚337 %loss/total_loss‚9.547821192384504 epoch‚10 0loss/FRt1/assignment_loss‚0.8259838319135074 2loss/SRqqt2/assignment_loss‚0.7091113821058932 2loss/FBt1/assignment_loss‚0.004187971076421542 /loss/FRt2/detection_loss‚0.5932862959735725 1loss/SRqqt2/detection_loss‚1.1149556993517236 /loss/FBt1/detection_loss‚0.8094932254707384 _runtime‚2749.817455013 0loss/FBt2/assignment_loss‚0.0866751856588271 /loss/FRt1/detection_loss‚1.5587914224124135 .loss/FBt2/detection_loss‚0.529580482716223 trainer/global_step‚4099 % _timestamp‚1.7844955932159123e+09ÑS„ïß9£‚(:® óûôÒÒر¿* proc.memory.percent‚18.093490600585938 cpu‚25.810579149312083 network.sent‚ 5.995084e+06% disk.nvme0c0n1.out‚16722.32421875 disk.nvme0n1.in‚0' disk.nvme1c1n1.out‚9587.74951171875 disk.dm-0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚759187% disk./.usageGB‚1171.4361686706543 disk.nvme0c0n1.in‚0# disk.nvme0n1.out‚16722.32421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" disk.dm-1.out‚2385.22119140625 proc.memory.rssMB‚ 2964.4375 disk.md0.in‚0" disk.dm-0.out‚23773.8740234375 disk.dm-1.in‚0 disk.md0.out‚ 70.00390625) disk./.usagePercent‚76.30263270937334% disk.nvme1n1.out‚9587.74951171875 disk.rbd0.in‚0$ memory_percent‚98.43893051147461& proc.memory.availableMB‚ 255.765625a:ÝŸ¤‚(:” óûôÒ™èã¿ gpu.0.gpu‚79.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚117.169' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.056333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ïÆ¿9¥‚(:® ‚üôÒ‚–á¾ cpu‚25.80927947647772( disk./.usagePercent‚76.3026379273276 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚16726.490234375 disk.nvme1c1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚23777.89208984375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚9588.9794921875 disk.md0.out‚ 70.609375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2806.00390625 network.sent‚ 6.046633e+06 disk.nvme1n1.in‚0" disk.dm-1.out‚2385.31103515625& proc.memory.availableMB‚ 266.734375 proc.cpu.threads‚17 network.recv‚763187% disk./.usageGB‚1171.4362487792969& disk.nvme0c0n1.out‚16726.490234375$ disk.nvme1n1.out‚9588.9794921875 disk.dm-0.in‚0$ memory_percent‚98.37198257446289) proc.memory.percent‚17.12648868560791³h›¦‚(:’ ‚üôÒȪ׿ gpu.0.gpu‚14.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚207.623' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚69.20766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸JLþ2§- _wandbruntime‚2780  _runtime‚2780ä,ÌoȨ _step‚338 ' lr-AdamW/pg1‚0.00014639652986964181 ' lr-AdamW/pg2‚0.00014639652986964181 trainer/global_step‚4149 $ _timestamp‚1.784495624276128e+09 _runtime‚2780.877954463ÛÉÜèÍ©Ç _runtime‚2780.877954463 _step‚338 ' lr-AdamW/pg1‚0.00014639652986964181 ' lr-AdamW/pg2‚0.00014639652986964181 trainer/global_step‚4149 $ _timestamp‚1.784495624276128e+09Ò:¾€2ª- _wandbruntime‚2780  _runtime‚2780±¯Wf«ÚO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321855157319168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segb]}z%ment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244572978364416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325276157771264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41728917 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32882323 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74621383 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7844956242770784e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325276263470592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚318058771359632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960e‘m›å'¬ßO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9394833 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31460895 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3522273469952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321470487995086 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321855174358528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32882321 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325276157771264 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41728917 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32882323 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74621383 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7844956242770784e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325276263470592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚318058771359632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321855157319168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244572978364416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325352198859776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9396814 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321545562015190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74638846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22071303 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241106713178624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31468296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41748894 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41738737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241106619068928 trainer/global_step‚4149 % _timestamp‚1.7844956246794841e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32889950 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41738737 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244629707261440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚318133078924384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32889950 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41748894 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74628687 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321930324024832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚2781.281472418 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74628687 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325352093153280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321930324024832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22071307 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3412483090806 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3421786167808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32889952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74638846 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31468117 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9396989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚08ì·b ¯éM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41748894 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74628687 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321930324024832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚2781.281472418 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74628687 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325352093153280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321930324024832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22071307 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3412483090806 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3421786167808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32889952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74638846 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31468117 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9396989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244629928710656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3412396304268 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32889952 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325352198859776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3421874834944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325352198859776 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚175 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9396814 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321545562015190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74638846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22071303 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241106713178624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31468296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41748894 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41738737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241106619068928 trainer/global_step‚4149 % _timestamp‚1.7844956246794841e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32889950 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚179 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41738737 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244629707261440 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚318133078924384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32889950 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329072991026688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42221769 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247420387354112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844956553860495e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31821630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31821449 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42231912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9502579 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3450938126902 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325612657673728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329073096726016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243857461850624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321773297629232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3450851340476 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33263771 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3460350392320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325224148969708 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75485540 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243857367740928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42221769 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33263771 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2811.989334584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247420165897728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚343 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4199 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33263773 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22319051 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321773314668592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22319047 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33263773 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325612657673728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75495685 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329073096726016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329072991026688 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4Qå'¼ßO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42221769 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247420387354112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844956553860495e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31821630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31821449 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42231912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚181 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9502579 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3450938126902 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325612657673728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329073096726016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243857461850624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321773297629232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3450851340476 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33263771 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3460350392320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325224148969708 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75485540 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243857367740928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42221769 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33263771 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2811.989334584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247420165897728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚343 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4199 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33263773 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22319051 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321773314668592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22319047 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33263773 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325612657673728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75495685 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329073096726016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329072991026688 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325612640634368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3460350392320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329072991026688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360×ÀÚ9š2½- _wandbruntime‚2812  _runtime‚2812MåYØ)¾äM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42241746 _runtime‚2812.390719407 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31828565 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325299326815598 E?DeviceStatsMonitor.on_train_baYüÁ$atch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42231589 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243913505468928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42241746 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3563610585600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33271402 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75513148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22324107 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75513148 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3451634835076 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22324103 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247477116054528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9504644 trainer/global_step‚4199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243913599578624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321847605193984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _step‚344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75502989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329148926408704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3563740022272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3461224775168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3461224775168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844956557878106e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325687807340032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚186 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325687807340032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33271400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329149032115200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42231589 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247477339600896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329149032115200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325687790300672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75502989 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33271400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33271402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3451721621614 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42231589 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243913505468928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42241746 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3563610585600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33271402 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75513148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22324107 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75513148 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3451634835076 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22324103 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247477116054528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9504644 trainer/global_step‚4199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243913599578624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321847605193984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 _step‚344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75502989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329148926408704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3563740022272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3461224775168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3461224775168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844956557878106e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325687807340032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚186 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325687807340032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33271400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329149032115200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42231589 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247477339600896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329149032115200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325687790300672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75502989 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33271400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33271402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3451721621614 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31828565 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325299326815598س7¨™2À- _wandbruntime‚2812  _runtime‚2812q±Â‘Á‚ /loss/FBt1/detection_loss‚0.5598799799262133 epoch‚10 % _timestamp‚1.7844956557887542e+09 2loss/SRqqt1/assignment_loss‚0.5605236384938189 2loss/SRqqt2/assignment_loss‚0.5824129735236458 1loss/SRqqt2/detection_loss‚0.9863934981069531 %loss/total_loss‚9.159372715504752 _step‚345 /loss/FRt2/assignment_loss‚2.082150529046293 /loss/FRt2/detection_loss‚0.5990263192728749 /loss/FBt2/detection_loss‚0.5743604314276595 trainer/global_step‚4199 0loss/FRt1/assignment_loss‚0.8487208110910234 _runtime‚2812.391026827 2loss/FBt1/assignment_loss‚0.008824601033283383 1loss/FBt2/assignment_loss‚0.16064781793087762 /loss/FRt1/detection_loss‚1.5786034992154054 1loss/SRqqt1/detection_loss‚0.6178286164367056iÓ9ü ‡ /loss/FRt1/detection_loss‚1.5786034992154054 1loss/SRqqt1/detection_loss‚0.6178286164367056 /loss/FBt1/detection_loss‚0.5598799799262133 epoch‚10 % _timestamp‚1.7844956557887542e+09 2loss/SRqqt1/assignment_loss‚0.5605236384938189 2loss/SRqqt2/assignment_loss‚0.5824129735236458 1loss/SRqqt2/detection_loss‚0.9863934981069531 %loss/total_loss‚9.159372715504752 _step‚345 /loss/FRt2/assignment_loss‚2.082150529046293 /loss/FRt2/detection_loss‚0.5990263192728749 /loss/FBt2/detection_loss‚0.5743604314276595 trainer/global_step‚4199 0loss/FRt1/assignment_loss‚0.8487208110910234 _runtime‚2812.391026827 2loss/FBt1/assignment_loss‚0.008824601033283383 1loss/FBt2/assignment_loss‚0.16064781793087762ÙlwÞ<Â(:± ¯üôÒþ倿 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 71.7265625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚16736.5673828125 disk.md0.in‚0 disk.dm-0.in‚0! disk.dm-0.out‚23787.865234375' proc.memory.availableMB‚ 266.3828125# proc.memory.rssMB‚ 2806.00390625 network.sent‚ 6.11661e+06" disk.dm-1.out‚2385.74169921875) proc.memory.percent‚17.12648868560791 cpu‚25.805775161306144$ disk./.usageGB‚1171.230209350586' disk.nvme0c0n1.out‚16736.5673828125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9591.74072265625# memory_percent‚98.3741283416748 proc.cpu.threads‚17' disk.nvme1c1n1.out‚9591.74072265625 network.recv‚774624) disk./.usagePercent‚76.28921734900413 disk.rbd0.in‚0 disk.rbd0.out‚0ðNÄ‚(:’ ¯üôÒ¯ä¿ gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚224.203' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.73433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q*G•CÅ‚(:¸ ¾üôÒÞÒä¾) disk./.usagePercent‚76.28922480322449$ disk./.usageGB‚1171.230323791504 disk.md0.out‚ 72.00390625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9592.20166015625% disk.nvme1n1.out‚9592.20166015625 network.sent‚ 6.180811e+06 disk.nvme1n1.in‚0$ memory_percent‚98.37291240692139( proc.memory.availableMB‚ 266.58203125" disk.dm-1.out‚2385.79638671875& disk.nvme0n1.out‚16741.28076171875# proc.memory.rssMB‚ 2806.00390625) proc.memory.percent‚17.12648868560791 cpu‚25.80460812809716 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚0# disk.dm-0.out‚23792.36767578125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚16741.28076171875 disk.nvme0n1.in‚0 network.recv‚778785Sz÷žÆ‚(:“ ¾üôÒœ”¿ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚225.689' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.22966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øŽöa2Ç- _wandbruntime‚2843  _runtime‚28433ë‰@ÆÈÀ $ _timestamp‚1.784495686687168e+09 _runtime‚2843.290262577 _step‚346 & lr-AdamW/pg1‚0.0001456610271083171 & lr-AdamW/pg2‚0.0001456610271083171 trainer/global_step‚4249ŒâiðËÉÅ & lr-AdamW/pg1‚0.0001456610271083171 & lr-AdamW/pg2‚0.0001456610271083171 trainer/global_step‚4249 $ _timestamp‚1.784495686687168e+09 _runtime‚2843.290262577 _step‚346Úzzº2Ê- _wandbruntime‚2843  _runtime‚2843,µu&ø!ËÚO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22571851 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76359842 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42724764 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328977913770116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325487840938192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844956866877356e+09 _step‚347 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32180988 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚4249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33645223 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329370140988928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33645221 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3490089871284 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32181182 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332869929981440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250267494107136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3490176657710 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329370123949568 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3499788992512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3499788992512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42714621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246664254140928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332869929981440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246664348250624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚194 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9609331 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329370123949568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76369987 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚190 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250267713466368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332869824282112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3603365215744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325487823898832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33645223 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328978017595902 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332869824282112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360hl"å'ÌßO % _timestamp‚1.7844956866877356e+09 _step‚347 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32180988 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚4249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33645223 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329370140988928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33645221 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3490089871284 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32181182 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332869929981440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250267494107136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3490176657710 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329370123949568 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3499788992512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3499788992512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42714621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246664254140928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332869929981440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246664348250624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚194 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9609331 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329370123949568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76369987 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚190 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250267713466368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332869824282112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3603365215744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325487823898832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33645223 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328978017595902 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332869824282112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22571851 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76359842 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42724764 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328977913770116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325487840938192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Û»Ð2Í- _wandbruntime‚2843  _runtime‚2843!ýäê&ÎäM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3500486048256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚186 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329053091616006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42734598 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33652850 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _step‚348 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22576907 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3604044194816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76387450 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚332945759664128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76387450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32188268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3490960152422 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332945865370624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9611179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250324661707264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚4249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22576903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332945865370624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3500574715392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3500486048256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329445290655232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329052987790108 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325562114424224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32188082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42734598 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3490873365884 _runtime‚2843.688898987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42724441 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7844956870869212e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42724441Sk2ï&ÏéM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33652850 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250324436063744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329445273615872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246720391868928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22576907 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3604044194816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76387450 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚332945759664128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76387450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32188268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3490960152422 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332945865370624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9611179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250324661707264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚4249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22576903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332945865370624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3500574715392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3500486048256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329445290655232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329052987790108 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325562114424224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32188082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42734598 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3490873365884 _runtime‚2843.688898987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42724441 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7844956870869212e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42724441 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3500486048256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚186 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329053091616006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42734598 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33652850 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _step‚348 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336666657537536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77234144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3539050272768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336666763236864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332731678570524 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333127624304128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9715929 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34026671 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3539050272768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43207473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333127607264768 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3529328402092 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43217616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34026673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22824647 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253114805810688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253115027267072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3643665269760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3529415188518 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32540580 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34026671 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249471234650624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329202367207792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3643792616448 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3529328402092 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43217616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34026673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22824647 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253114805810688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253115027267072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3643665269760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3529415188518 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32540580 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34026671 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249471234650624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329202367207792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3643792616448 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336666657537536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77234144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3539050272768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336666763236864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332731678570524 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333127624304128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9715929 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34026671 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3539050272768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ßì…|²2Ý- _wandbruntime‚2874  _runtime‚2874vƒœò¸!ÞäM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22829707 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332806856416414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336742592919552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9717894 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336742698626048 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77261752 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333202773970432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32547787 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43217293 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 _runtime‚2874.961216627 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333202756931072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43217293 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333202773970432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43227450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249527278268928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332806752590516 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336742698626048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253171757745152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3644479476224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3530111896692 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844957183591764e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32547597 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77251593 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3539924655616 \JDe¢N¾›2viceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3539924655616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22829703 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249527372378624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43227450 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539835988480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34034302 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329276640693824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175-xnï&ßéM YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32547787 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43217293 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 _runtime‚2874.961216627 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333202756931072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43217293 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333202773970432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43227450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249527278268928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332806752590516 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336742698626048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253171757745152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3644479476224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3530111896692 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844957183591764e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32547597 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77251593 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3539924655616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3539924655616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22829703 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249527372378624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43227450 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539835988480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34034302 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329276640693824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22829707 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332806856416414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336742592919552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9717894 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336742698626048 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77261752 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333202773970432 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336885090579968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43700325 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34408121 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255962199251968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336485547196718 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23077451 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336885107619328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 trainer/global_step‚4349 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32899141 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684301754880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43700325 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34408121 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34408123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32899333 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9821694 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23077447 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34408123 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3578488872960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78108446 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340463490792960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336885090579968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43700325 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34408121 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255962199251968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336485547196718 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23077451 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336885107619328 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 trainer/global_step‚4349 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32899141 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684301754880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43700325 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34408121 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34408123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32899333 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9821694 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23077447 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34408123 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3578488872960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78108446 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340463490792960 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78125895 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9823972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256019366589952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23082503 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78136054 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336560621216822 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3579185928704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚183 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332991166963424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336960257285632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78136054 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252334258778624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3569437214038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32906296 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336960257285632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34415752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32906479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43710145 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚340539426174976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43720302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340539531881472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23082507 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 % _timestamp‚1.7844957483201828e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34415750 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336560517390924 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3579185928704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 2904.92215827 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34415752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9823793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0UW‚î&ïèM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336960240246272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252334164668928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3569350427500 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340539531881472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78125895 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9823972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256019366589952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23082503 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78136054 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336560621216822 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3579185928704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚183 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332991166963424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336960257285632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78136054 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252334258778624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3569437214038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32906296 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336960257285632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34415752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32906479 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43710145 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚340539426174976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43720302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340539531881472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23082507 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 % _timestamp‚1.7844957483201828e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34415750 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336560517390924 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3579185928704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚ 2904.92215827 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34415752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9823793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256019143043584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344260324048384 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33257586 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340642573895168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3617838813184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3617750153216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340239208171340 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9927339 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚4399 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚177 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3617750153216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78992893 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44203320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258809568570368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 36078„–u”‡05463708 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23330247 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34789573 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9927512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3724786770432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255085007450624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44203320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258809794221056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255085007450624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44203320 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258809794221056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344260324048384 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33257586 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340642573895168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3617838813184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3617750153216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340239208171340 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9927339 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚4399çÏÿm*2ý- _wandbruntime‚2935  _runtime‚2935ëxM|é&þãM PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340717723561472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _step‚360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33264787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231934976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340717740600832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3608675744846 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3608588958308 $ _timestamp‚1.784495778539725e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23335303 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3618624536064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2935.141691383 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44202997 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3725598733312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33264971 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340314386017230 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79000197 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255141145178624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚336705693233024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3618535868928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9929664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137825280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3618535868928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34797202 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9929484 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23335307 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258866743911936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44213154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34797202 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚336705710272384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34797200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚344336365136896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344336259430400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79010356 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44213154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3725460908032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340717740600832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3618624536064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340314282191332 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79010356 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344336365136896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚340717723561472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34797200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33264971 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340314386017230 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79000197 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255141145178624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚336705693233024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3618535868928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9929664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137825280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3618535868928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34797202 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9929484 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23335307 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258866743911936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44213154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34797202 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚336705710272384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34797200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚344336365136896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344336259430400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79010356 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44213154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3725460908032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340717740600832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3618624536064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340314282191332 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79010356 ;4DeviceStatsMonitor.o>“Sè— n_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344336365136896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚340717723561472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34797200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340717723561472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 _step‚360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33264787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231934976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340717740600832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3608675744846 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3608588958308 $ _timestamp‚1.784495778539725e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23335303 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3618624536064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2935.141691383 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130èK5ëÂ2€- _wandbruntime‚2935  _runtime‚2935¾QÕ‚ /loss/FRt2/assignment_loss‚2.089350939086059 2loss/SRqqt1/assignment_loss‚0.6716129024868125 %loss/total_loss‚9.659383093443942 epoch‚10 0loss/FRt1/assignment_loss‚0.9027906469611883 2loss/SRqqt2/assignment_loss‚0.7016716439024427 /loss/FRt2/detection_loss‚0.5711525808633282 1loss/SRqqt1/detection_loss‚0.6496032781590458 /loss/FBt2/detection_loss‚0.5867069299216382 % _timestamp‚1.7844957785406806e+09 2loss/FBt1/assignment_loss‚0.003931132888383928 /loss/FBt1/detection_loss‚0.8153018014805276 _step‚361 1loss/FBt2/assignment_loss‚0.15879357999486274 /loss/FRt1/detection_loss‚1.4342985647469992 1loss/SRqqt2/detection_loss‚1.0741690929526542 trainer/global_step‚4399 _runtime‚2935.142207674)É 2 ‚‡ 1loss/FBt2/assignment_loss‚0.15879357999486274 /loss/FRt1/detection_loss‚1.4342985647469992 1loss/SRqqt2/detection_loss‚1.0741690929526542 trainer/global_step‚4399 _runtime‚2935.142207674 /loss/FRt2/assignment_loss‚2.089350939086059 2loss/SRqqt1/assignment_loss‚0.6716129024868125 %loss/total_loss‚9.659383093443942 epoch‚10 0loss/FRt1/assignment_loss‚0.9027906469611883 2loss/SRqqt2/assignment_loss‚0.7016716439024427 /loss/FRt2/detection_loss‚0.5711525808633282 1loss/SRqqt1/detection_loss‚0.6496032781590458 /loss/FBt2/detection_loss‚0.5867069299216382 % _timestamp‚1.7844957785406806e+09 2loss/FBt1/assignment_loss‚0.003931132888383928 /loss/FBt1/detection_loss‚0.8153018014805276 _step‚361éÑ‹KAƒ‚(:¶ §ýôÒàÒÖ¿ network.sent‚ 6.398246e+06 network.recv‚804878) disk./.usagePercent‚76.26587296712478# disk.dm-0.out‚24159.90966796875 disk.nvme1n1.in‚0 disk.md0.out‚ 74.95703125$ memory_percent‚98.86770248413086 cpu‚25.802566046066406" disk.dm-1.out‚2386.88916015625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚9627.29931640625 disk.md0.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1170.8718147277832 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚9627.29931640625 proc.memory.rssMB‚ 2886.34375 disk.dm-0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚17081.13232421875& disk.nvme0n1.out‚17081.13232421875& proc.memory.availableMB‚ 185.515625* proc.memory.percent‚17.616844177246094îô‘;“„‚(:ˆ §ýôÒÜ™¿ gpu.0.gpu‚86.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚220.257' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÆó2…- _wandbruntime‚2948  _runtime‚2948²l=²=†¬{ 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6257093548774719 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7242441773414612 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1911574751138687 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7598192691802979 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2565062344074249 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36992934346199036 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7195476293563843 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6520146131515503 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9785090088844299 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6191444993019104 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0377596840262413 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.640625 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8163266181945801 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8254724144935608 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18040059506893158 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6227197051048279 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8301209807395935 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20344577729701996 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_2‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14454612135887146 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007558579673059285 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7680460810661316 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1836947202682495 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19114772975444794 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06787054985761642 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03372130170464516 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16595244407653809 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 trainer/global_step‚4399 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 jet/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6624634861946106 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.39422667026519775 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6078287959098816 (particle/accuracy‚0.8474547863006592 )loss/val_total_loss‚9.095924277675122 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47186973690986633 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.6277220845222473 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872875213623047 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909443855285645 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6404959559440613 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.691754937171936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6247562766075134 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6050515174865723 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5688363909721375 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11970708519220352 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08607766032218933 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 4 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940185546875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9925707578659058 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938130974769592 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7302720546722412 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 .particle/accuracy_4_of_4‚0.939069926738739 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17095886170864105 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5470833778381348 *jet/accuracy_2_of_3‚0.5072823762893677 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11127470433712006 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2093929946422577 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7612617611885071 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26004016399383545 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9171833992004395 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07056643813848495 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.612521767616272 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.797902524471283 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6386382579803467 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970513582229614 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1652459055185318 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16503804922103882 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1911574751138687 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7680460810661316 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0704219788312912 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36857694387435913 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6679491996765137 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_1‚0.3141099512577057 +jet/accuracy_3_of_3‚0.07071267813444138 *jet/accuracy_2_of_4‚0.7114170789718628 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5738883018493652 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027094585821032524 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6965217590332031 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16314542293548584 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47337809205055237 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5960339903831482 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 % _timestamp‚1.7844957916523292e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950168132781982 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8510090708732605 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3825596272945404 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_2‚0.7769099473953247 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8581147789955139 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9818164110183716 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5622968673706055 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009418776258826256 *jet/accuracy_1_of_3‚0.9141219258308411 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5557481646537781 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6309829354286194 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843841791152954 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6301785707473755 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2759164869785309 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740740299224854 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5209572911262512 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6validation_average_jet_accuracy‚0.3567672073841095 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7332273125648499 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1857907921075821 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6190058588981628 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0007558579673059285 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06871633976697922 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.763750433921814 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2046772837638855 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46645399928092957 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚362 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2975656986236572 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6301785707473755 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 epoch‚10 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7613465785980225 particle/accuracy_1_of_3‚1 jet/accuracy_3_of_5‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01766383834183216 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8183674812316895 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16499999165534973 +particle/specificity‚0.9075471758842468 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8956095576286316 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6735491752624512 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_2‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9654542207717896 particle/accuracy_4_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923298358917236 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10252106934785843 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.668657660484314 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.675635814666748 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17423707246780396 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9126515984535217 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _runtime‚2948.254573352 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4129420220851898 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017663011327385902 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6708064079284668 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6272896528244019 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5471315979957581 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0704219788312912 +jet/accuracy_2_of_2‚0.13716325163841248 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977208971977234 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1787574589252472 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7324107885360718 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15556283295154572 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2041110247373581 particle/accuracy_2_of_3‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6225108504295349 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.987639844417572 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17851261794567108 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0710776224732399 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9687683582305908 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997998833656311 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.129057377576828 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3680555522441864 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7281625866889954 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18966953456401825 particle/accuracy_2_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344^²§Ë+*‡±{ epoch‚10 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7613465785980225 particle/accuracy_1_of_3‚1 jet/accuracy_3_of_5‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01766383834183216 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8183674812316895 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16499999165534973 +particle/specificity‚0.9075471758842468 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8956095576286316 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6735491752624512 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740740299224854 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_2‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9654542207717896 particle/accuracy_4_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923298358917236 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10252106934785843 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.668657660484314 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.675635814666748 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17423707246780396 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9126515984535217 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _runtime‚2948.254573352 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4129420220851898 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017663011327385902 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6708064079284668 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6272896528244019 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5471315979957581 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0704219788312912 +jet/accuracy_2_of_2‚0.13716325163841248 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977208971977234 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1787574589252472 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7324107885360718 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15556283295154572 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2041110247373581 particle/accuracy_2_of_3‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6225108504295349 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.987639844417572 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17851261794567108 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0710776224732399 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9687683582305908 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997998833656311 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.129057377576828 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3680555522441864 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7281625866889954 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18966953456401825 particle/accuracy_2_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6257093548774719 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7242441773414612 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1911574751138687 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7598192691802979 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2565062344074249 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36992934346199036 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7195476293563843 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6520146131515503 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9785090088844299 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6191444993019104 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0377596840262413 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.640625 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8163266181945801 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8254724144935608 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18040059506893158 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6227197051048279 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8301209807395935 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20344577729701996 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_2‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14454612135887146 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007558579673059285 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7680460810661316 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1836947202682495 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19114772975444794 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06787054985761642 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03372130170464516 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16595244407653809 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 trainer/global_step‚4399 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 jet/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6624634861946106 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.39422667026519775 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6078287959098816 (particle/accuracy‚0.8474547863006592 )loss/val_total_loss‚9.095924277675122 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47186973690986633 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.6277220845222473 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872875213623047 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909443855285645 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6404959559440613 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.691754937171936 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6247562766075134 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6050515174865723 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5688363909721375 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11970708519220352 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08607766032218933 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 4 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940185546875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9925707578659058 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938130974769592 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7302720546722412 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 .particle/accuracy_4_of_4‚0.939069926738739 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17095886170864105 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5470833778381348 *jet/accuracy_2_of_3‚0.5072823762893677 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11127470433712006 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2093929946422577 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7612617611885071 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26004016399383545 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9171833992004395 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07056643813848495 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.612521767616272 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.79790252447l7-Œ1283 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6386382579803467 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970513582229614 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1652459055185318 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16503804922103882 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1911574751138687 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7680460810661316 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0704219788312912 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36857694387435913 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6679491996765137 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_1‚0.3141099512577057 +jet/accuracy_3_of_3‚0.07071267813444138 *jet/accuracy_2_of_4‚0.7114170789718628 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5738883018493652 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027094585821032524 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6965217590332031 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16314542293548584 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47337809205055237 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5960339903831482 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 % _timestamp‚1.7844957916523292e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950168132781982 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8510090708732605 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3825596272945404 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_2‚0.7769099473953247 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8581147789955139 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9818164110183716 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5622968673706055 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009418776258826256 *jet/accuracy_1_of_3‚0.9141219258308411 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5557481646537781 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6309829354286194 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843841791152954 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6301785707473755 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2759164869785309 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740740299224854 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5209572911262512 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6validation_average_jet_accuracy‚0.3567672073841095 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7332273125648499 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1857907921075821 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6190058588981628 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0007558579673059285 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06871633976697922 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.763750433921814 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2046772837638855 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46645399928092957 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚362 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2975656986236572 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6301785707473755 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1êçÕ÷Ù6ˆ‚(:« ¶ýôÒÕñ¾( disk.nvme0c0n1.out‚17127.88232421875 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 network.sent‚ 6.446963e+06$ disk./.usageGB‚1170.871883392334 disk.rbd0.out‚0$ disk.nvme1n1.out‚9629.6396484375$ memory_percent‚98.86703491210938 proc.memory.rssMB‚ 2886.34375* proc.memory.percent‚17.616844177246094 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚9629.6396484375 disk.md0.out‚75.40625" disk.dm-1.out‚2387.01025390625& disk.nvme0n1.out‚17127.88232421875" disk.dm-0.out‚24207.9091796875# proc.memory.availableMB‚185.625 cpu‚25.80997760468124 proc.cpu.threads‚17 network.recv‚809464) disk./.usagePercent‚76.26587743965699 u΂Ÿ‰‚(:” ¶ýôÒ·«é¿ gpu.0.gpu‚56.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚157.357' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚52.452333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðë1Š‚(:¦ ÅýôÒ¯˜¿ network.sent‚ 6.505756e+06( disk./.usagePercent‚76.0888397062743% disk./.usageGB‚1168.1539115905762 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 185.61328125 proc.memory.rssMB‚ 2886.34375 network.recv‚813850 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚17128.6923828125 disk.dm-0.in‚0* proc.memory.percent‚17.616844177246094 cpu‚25.80883436687472' disk.nvme0c0n1.out‚17128.6923828125" disk.nvme1n1.out‚ 9630.15234375 disk.md0.in‚0 disk.md0.out‚75.6875# memory_percent‚98.8671064376831 proc.cpu.threads‚17 disk.dm-1.out‚2387.103515625 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 9630.15234375" disk.dm-0.out‚24208.51855468759"á ž‹‚(:“ ÅýôÒ®ö¿¿ gpu.0.gpu‚77.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚231.941' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.31366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uUŠ2Œ- _wandbruntime‚2978  _runtime‚2978OƇjÉà trainer/global_step‚4449 % _timestamp‚1.7844958223457358e+09 _runtime‚2978.947577517 _step‚363 ' lr-AdamW/pg1‚0.00014413978175805388 ' lr-AdamW/pg2‚0.00014413978175805388¶²ÎŽÈ ' lr-AdamW/pg1‚0.00014413978175805388 ' lr-AdamW/pg2‚0.00014413978175805388 trainer/global_step‚4449 % _timestamp‚1.7844958223457358e+09 _runtime‚2978.947577517 _step‚363ë"Ç2- _wandbruntime‚2978  _runtime‚2978Sn3Æâ'ÜO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35292821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262476199918080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344981959092287 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚345396561545728 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚4449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44778674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44768531 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662459253077 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349069073974272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44778674 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341319413052784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚364 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3780819807232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3672618127872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349069179673600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44768531 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚345396561545728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚258695259055616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80061352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35292823 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344981855266501 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33772998 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚2978.952274484 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844958223466635e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23691306 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349069073974272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35292823 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345396544506368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262475984753152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345396544506368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3662546039503 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80071497 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚23tÝ‹ç'‘áO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80061352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35292823 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344981855266501 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33772998 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 _runtime‚2978.952274484 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844958223466635e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23691306 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349069073974272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35292823 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚345396544506368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262475984753152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345396544506368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3662546039503 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80071497 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35292821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262476199918080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344981959092287 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚345396561545728 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚4449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44778674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44768531 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662459253077 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349069073974272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44778674 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341319413052784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚364 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3780819807232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3672618127872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349069179673600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44768531 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚345396561545728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚258695259055616ì.€32’- _wandbruntime‚2979  _runtime‚2979ïeø.“æM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345057033112391 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262532936540160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3663242747677 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258751396783616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80078801 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35300452 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349145115062784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚345471694172672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44788508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚168 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33780203 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚365 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349145115062784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3673315183616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35300450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚164 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262533143309312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341393686538816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80088960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663329534215 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10084009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 trainer/global_step‚4449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345471694172672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345471711212032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23696362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35300452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3673315183616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345471711212032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23696358 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3781633866240¸5U×ñ&”ëM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3663329534215 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10084009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 trainer/global_step‚4449 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345471694172672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345471711212032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23696362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35300452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3673315183616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345471711212032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23696358 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3781633866240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345057033112391 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262532936540160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3663242747677 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258751396783616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80078801 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35300452 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349145115062784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚345471694172672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44788508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚168 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33780203 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚365 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349145115062784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3673315183616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35300450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚164 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262533143309312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341393686538816 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80088960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812íÕÀê)2•- _wandbruntime‚2979  _runtime‚2979Ä]–‚ 1loss/SRqqt2/detection_loss‚0.9817712858784944 _step‚366 0loss/FRt1/assignment_loss‚0.8906366721228445 2loss/SRqqt2/assignment_loss‚0.6100158266142642 2loss/FBt1/assignment_loss‚0.004507720963337964 .loss/FBt2/detection_loss‚1.040275280068342 %loss/total_loss‚9.758395500400052 _runtime‚2979.346213173 0loss/FRt2/assignment_loss‚2.1067388975468186 2loss/SRqqt1/assignment_loss‚0.5934334966386916 epoch‚11 trainer/global_step‚4449 1loss/FBt2/assignment_loss‚0.11284409805050735 /loss/FRt1/detection_loss‚1.5150483421318588 /loss/FRt2/detection_loss‚0.5731255837239734 1loss/SRqqt1/detection_loss‚0.6727338383243681 /loss/FBt1/detection_loss‚0.6572644583365509 % _timestamp‚1.7844958227446582e+09ÿéOš —‡ 1loss/FBt2/assignment_loss‚0.11284409805050735 /loss/FRt1/detection_loss‚1.5150483421318588 /loss/FRt2/detection_loss‚0.5731255837239734 1loss/SRqqt1/detection_loss‚0.6727338383243681 /loss/FBt1/detection_loss‚0.6572644583365509 % _timestamp‚1.7844958227446582e+09 1loss/SRqqt2/detection_loss‚0.9817712858784944 _step‚366 0loss/FRt1/assignment_loss‚0.8906366721228445 2loss/SRqqt2/assignment_loss‚0.6100158266142642 2loss/FBt1/assignment_loss‚0.004507720963337964 .loss/FBt2/detection_loss‚1.040275280068342 %loss/total_loss‚9.758395500400052 _runtime‚2979.346213173 0loss/FRt2/assignment_loss‚2.1067388975468186 2loss/SRqqt1/assignment_loss‚0.5934334966386916 epoch‚11 trainer/global_step‚4449îQ5b:˜‚(:¯ ÔýôÒÿÝ·¿* proc.memory.percent‚17.616844177246094 proc.cpu.threads‚17 disk.dm-1.out‚2387.267578125) disk./.usagePercent‚76.12188351967691 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚17419.26904296875' proc.memory.availableMB‚ 185.5703125 network.sent‚ 6.521516e+06 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚9667.3310546875 disk.rbd0.out‚0 proc.memory.rssMB‚ 2886.34375 cpu‚25.80770044583979 network.recv‚817295% disk./.usageGB‚1168.6612167358398 disk.md0.in‚0 disk.md0.out‚76.28125 disk.dm-0.in‚0# disk.dm-0.out‚24534.85400390625 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚9667.3310546875( disk.nvme0c0n1.out‚17419.26904296875$ memory_percent‚98.86736869812012‘˜¥i“™‚(:ˆ ÔýôÒ‘÷ô¿ gpu.0.gpu‚92.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.845' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚75.615 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0opÌç@š‚(:µ ãýôңÅ¿ disk.md0.in‚0 disk.md0.out‚ 76.63671875 network.recv‚821738 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 185.578125* proc.memory.percent‚17.616844177246094 network.sent‚ 6.572781e+06% disk./.usageGB‚1168.6611061096191 disk.dm-0.in‚0# disk.dm-0.out‚24535.37353515625 disk.rbd0.out‚0' disk.nvme1c1n1.out‚9667.98095703125 proc.memory.rssMB‚ 2886.34375 cpu‚25.806590829917525 disk.rbd0.in‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚17420.06787109375% disk.nvme1n1.out‚9667.98095703125# memory_percent‚98.8673210144043 proc.cpu.threads‚17( disk.nvme0c0n1.out‚17420.06787109375) disk./.usagePercent‚76.12187631393057" disk.dm-1.out‚2387.40380859375 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0(£Ûìž›‚(:“ ãýôÒšŽ³¿ gpu.0.gpu‚84.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.839' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.27966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹f†£2œ- _wandbruntime‚3009  _runtime‚3009á¦Þ…Éà ' lr-AdamW/pg1‚0.00014374914588870876 ' lr-AdamW/pg2‚0.00014374914588870876 trainer/global_step‚4499 % _timestamp‚1.7844958530822072e+09 _runtime‚3009.684011423 _step‚367¶Uå9ÎžÈ % _timestamp‚1.7844958530822072e+09 _runtime‚3009.684011423 _step‚367 ' lr-AdamW/pg1‚0.00014374914588870876 ' lr-AdamW/pg2‚0.00014374914588870876 trainer/global_step‚4499ïÍé2Ÿ- _wandbruntime‚3009  _runtime‚3009b×å®â' ÜO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352866012929024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348735723892695 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265323527452160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35674273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3821476106240 trainer/global_step‚4499 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45261383 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345033939322384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10189961 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _step‚368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3711879408128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34134067 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352865907229696 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3711968068096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80945799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265323734228480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261502051345920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45271526 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3701784570311 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345033922283024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348735620066909 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34133890 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206776320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 206776320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352865907229696 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3711968068096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80945799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265323734228480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261502051345920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45271526 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3701784570311 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345033922283024ð`Câ2¢- _wandbruntime‚3010  _runtime‚3010)>Dÿì&£æM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349229194527232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352941842611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261558189073920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35681900 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚172 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚4499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34140981 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45281360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3822290140672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348810797912799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3702568065023 _step‚369 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345108212808416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80963262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35681902 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3822402800128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80953103 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352941948318208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3712753790976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349229194527232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3712665123840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345108229847776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3712665123840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3712753790976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261558283183616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349229177487872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34141153 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3702481278485 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 % _timestamp‚1.7844958534827302e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23949158 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45281360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352941948318208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10191823 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35681900 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45271203 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265380479214592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚3010.084746497 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80963262 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80953103 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35681902 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349229194527232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352941842611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261558189073920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35681900 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚172 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 trainer/global_step‚4499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34140981 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45281360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 206769152 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3822290140672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348810797912799 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3702568065023 _step‚369 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 112659456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345108212808416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80963262 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35681902 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3822402800128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80953103 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352941948318208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3712753790976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349229194527232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3712665123840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345108229847776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3712665123840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3712753790976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261558283183616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349229177487872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34141153 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3702481278485 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 % _timestamp‚1.7844958534827302e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23949158 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45281360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352941948318208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10191823 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35681900 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45271203 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265380479214592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚3010.084746497 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80963262 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80953103 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35681902 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 F¨‚(:³ òýôÒáóœ¿ disk.nvme0c0n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.616844177246094 disk.nvme1n1.in‚0 network.sent‚ 6.588588e+06) disk./.usagePercent‚76.11981621589902% disk.nvme1c1n1.out‚9684.302734375! disk.dm-1.out‚2387.5810546875( proc.memory.availableMB‚ 185.69921875 cpu‚25.805647026803722 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚17611.65185546875 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚9684.302734375 disk.md0.in‚0# disk.dm-0.out‚24737.07080078125 proc.memory.rssMB‚ 2886.34375 network.recv‚824815% disk./.usageGB‚1168.6294784545898 disk.rbd0.in‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚17611.65185546875 disk.md0.out‚ 79.61328125$ memory_percent‚98.86658191680908KäÐœ©‚(:‘ òýôÒË…Ê¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚215.065' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.68833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—iÑ<ª‚(:± þôÒÕ”ƒ¿ disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9700.62158203125 disk.dm-0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.86658191680908* proc.memory.percent‚17.616844177246094 cpu‚25.804555066712947 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚17647.462890625 disk.nvme1n1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 185.69921875 proc.cpu.threads‚17 network.sent‚ 6.63988e+06 network.recv‚828951$ disk.nvme0n1.out‚17647.462890625% disk.nvme1n1.out‚9700.62158203125 disk.md0.in‚0 disk.md0.out‚ 79.91015625 disk.dm-1.in‚0 proc.memory.rssMB‚ 2886.34375$ disk./.usageGB‚1170.261001586914 disk.nvme0n1.in‚0" disk.dm-0.out‚24773.2724609375" disk.dm-1.out‚2402.87646484375) disk./.usagePercent‚76.22608705988692ì+S®ž«‚(:“ þôÒ쟰¿ gpu.0.gpu‚92.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚151.142' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚50.38066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â&2¬- _wandbruntime‚3040  _runtime‚30407«éÉ­Ã ' lr-AdamW/pg1‚0.00014335442998982453 ' lr-AdamW/pg2‚0.00014335442998982453 trainer/global_step‚4549 % _timestamp‚1.7844958842135894e+09 _runtime‚3040.817263918 _step‚371ÉZ%Î®È trainer/global_step‚4549 % _timestamp‚1.7844958842135894e+09 _runtime‚3040.817263918 _step‚371 ' lr-AdamW/pg1‚0.00014335442998982453 ' lr-AdamW/pg2‚0.00014335442998982453óäç[l2¯- _wandbruntime‚3040  _runtime‚3040l62¨°ÛO N@Device…ÂÿçÑ'StatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356662740485120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352911511136768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3741023101119 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3740936314693 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352911528176128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356662740485120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3862211206144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚352489384867317 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24196906 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3862088053760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45764378 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356662846184448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264309031855616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268171243061760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352489488693103 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36055721 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3751229348352 trainer/global_step‚4549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36055723 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34494293 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348748448552624 _runtime‚3040.817347668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45754235 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352911528176128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 $ _timestamp‚1.784495884214208e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36055721 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45764378 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24196902 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3751318008320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34494116 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45754235 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268171025799680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348748465591984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3751318008320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81820101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3751229348352 trainer/global_step‚4549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36055723 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34494293 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348748448552624 _runtime‚3040.817347668 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45754235 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352911528176128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 $ _timestamp‚1.784495884214208e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36055721 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45764378 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24196902 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3751318008320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34494116 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45754235 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268171025799680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348748465591984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3751318008320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81820101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356662740485120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352911511136768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3741023101119 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3740936314693 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352911528176128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356662740485120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3862211206144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚352489384867317 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24196906 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3862088053760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45764378 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356662846184448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264309031855616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268171243061760ôËÄȘ2²- _wandbruntime‚3041  _runtime‚3041·Ñ–9ë&³åM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356738675867136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3752103731200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚373 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352986677842432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352986660803072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 $ _timestamp‚1.784495884612514e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81837564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352564562713207 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45764055 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10299388 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚4549 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚3041.214469727 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352986677842432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81827405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81837564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36063352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264365075473920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3741719809293 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3752015064064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24201958 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268227969329152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3752103731200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24201962 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356738781573632 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36063350 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10299571 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚352564458887309 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34501346 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840-¼• ´êM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚3041.214469727 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352986677842432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81827405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81837564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36063352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264365075473920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3741719809293 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3752015064064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24201958 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268227969329152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3752103731200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚1237 ¢Iç4 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24201962 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356738781573632 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36063350 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10299571 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚352564458887309 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34501346 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356738675867136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3752103731200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚373 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352986677842432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352986660803072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 $ _timestamp‚1.784495884612514e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81837564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚187 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352564562713207 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45764055 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚183 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10299388 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚4549õ0¢‚2µ- _wandbruntime‚3041  _runtime‚3041‘žýB¶‚ /loss/FBt2/detection_loss‚0.7907097385174533 trainer/global_step‚4549 % _timestamp‚1.7844958846134691e+09 _step‚374 2loss/SRqqt2/assignment_loss‚0.7182045014034135 1loss/SRqqt1/detection_loss‚0.6162603394971938 epoch‚11 /loss/FRt2/detection_loss‚0.5070029641003476 _runtime‚3041.214967998 /loss/FRt1/assignment_loss‚0.870818622621316 3loss/FBt1/assignment_loss‚0.0022506169345155656 /loss/FRt1/detection_loss‚1.5221777722941536 /loss/FBt1/detection_loss‚0.5147001300845274 %loss/total_loss‚9.569012504685364 /loss/FRt2/assignment_loss‚2.147216078144821 2loss/SRqqt1/assignment_loss‚0.6174374445480257 1loss/FBt2/assignment_loss‚0.17587123482489958 1loss/SRqqt2/detection_loss‚1.0863630617146982 ¯QÙ ·‡ /loss/FRt2/detection_loss‚0.5070029641003476 _runtime‚3041.214967998 /loss/FRt1/assignment_loss‚0.870818622621316 3loss/FBt1/assignment_loss‚0.0022506169345155656 /loss/FRt1/detection_loss‚1.5221777722941536 /loss/FBt1/detection_loss‚0.5147001300845274 %loss/total_loss‚9.569012504685364 /loss/FRt2/assignment_loss‚2.147216078144821 2loss/SRqqt1/assignment_loss‚0.6174374445480257 1loss/FBt2/assignment_loss‚0.17587123482489958 1loss/SRqqt2/detection_loss‚1.0863630617146982 /loss/FBt2/detection_loss‚0.7907097385174533 trainer/global_step‚4549 % _timestamp‚1.7844958846134691e+09 _step‚374 2loss/SRqqt2/assignment_loss‚0.7182045014034135 1loss/SRqqt1/detection_loss‚0.6162603394971938 epoch‚11ö™Z,><¸‚(:± þôÒÀô³¿$ disk./.usageGB‚1171.153995513916 disk.md0.out‚ 80.359375$ memory_percent‚98.86853694915771( proc.memory.availableMB‚ 185.37890625* proc.memory.percent‚17.616844177246094 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚19550.9365234375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚9705.29150390625 disk.md0.in‚0! disk.dm-0.out‚26680.365234375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚832284) disk./.usagePercent‚76.28425308672307 disk.nvme0n1.in‚0% disk.nvme0n1.out‚19550.9365234375 disk.dm-0.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2886.34375 cpu‚25.803463533071408' disk.nvme1c1n1.out‚9705.29150390625 disk.nvme1n1.in‚0" disk.dm-1.out‚2402.95458984375 disk.rbd0.out‚0 network.sent‚ 6.655645e+06 ¹œ¹‚(:‘ þôÒ×úà¿ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚224.462' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.82066666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08 ã6º‚(:« ŸþôÒçÄò¾ disk.rbd0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚9706.20458984375 disk.dm-0.out‚26681.82421875$ memory_percent‚98.86705875396729 cpu‚25.802316951702142 network.sent‚ 6.707011e+06$ disk./.usageGB‚1171.154182434082" disk.dm-1.out‚2403.38818359375" disk.nvme0n1.out‚ 19552.7265625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9706.20458984375 disk.md0.in‚0( proc.memory.availableMB‚ 185.62109375) disk./.usagePercent‚76.28426526194964 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 19552.7265625 disk.nvme0n1.in‚0 disk.md0.out‚ 80.7421875 disk.dm-0.in‚0 proc.memory.rssMB‚ 2886.34375* proc.memory.percent‚17.616844177246094 network.recv‚836590 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17°#”ž»‚(:“ ŸþôÒÏ´ ¿ gpu.0.gpu‚92.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚234.372' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.12400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mŠŽÊ2¼- _wandbruntime‚3072  _runtime‚3072„EäôȽ ' lr-AdamW/pg1‚0.00014295565941326577 ' lr-AdamW/pg2‚0.00014295565941326577 trainer/global_step‚4599 % _timestamp‚1.7844959155292974e+09 _runtime‚ 3072.13225559 _step‚375°ë®‹Í¾Ç % _timestamp‚1.7844959155292974e+09 _runtime‚ 3072.13225559 _step‚375 ' lr-AdamW/pg1‚0.00014295565941326577 ' lr-AdamW/pg2‚0.00014295565941326577 trainer/global_step‚4599÷ý€2¿- _wandbruntime‚3072  _runtime‚3072 ,Äà'ÀÚO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82694403 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 3072.1323136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352462991861584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3780174845501 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36437173 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36437171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360459679439872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _step‚376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46257230 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚356669011491328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3790667948544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36437171 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360459679439872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36437173 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10405828 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3902763313152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3902640160768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356668994451968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844959155299935e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10406008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356243149667725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356243253493511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3780261631927 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360459573740544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267115824145920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360459573740544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚4599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82684258 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3790579288576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356669011491328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34855530 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271018681568768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46247087 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46257230 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24449702 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46247087 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328Pð¬ å'ÁßO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36437171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360459679439872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 _step‚376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46257230 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚356669011491328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3790667948544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36437171 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360459679439872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36437173 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10405828 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3902763313152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3902640160768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356668994451968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844959155299935e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10406008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356243149667725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356243253493511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3780261631927 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360459573740544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267115824145920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360459573740544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚4599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82684258 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3790579288576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356669011491328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34855530 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271018681568768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46247087 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46257230 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24449702 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46247087 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82694403 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 3072.1323136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352462991861584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3780174845501 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36437173øj†ƒ2Â- _wandbruntime‚3072  _runtime‚3072.“ò¥ì&ÃæM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82701707 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24454758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3903573408256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360535614829056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360535614829056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271075412136960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356318327513615 _step‚377 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271075629391872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352537265347616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356744144118272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46256907 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3791365004288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267172055983616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3791453671424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82711866 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34862729 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3780958340101 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _runtime‚3072.533361615 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82701707 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360535509122560 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10408155 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3903450263040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 % _timestamp‚1.7844959159313974e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46256907 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚188 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82711866 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34862917 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46267064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚4599†©tñ&ÄëM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82711866 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34862729 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3780958340101 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _runtime‚3072.533361615 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82701707 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360535509122560 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10408155 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3903450263040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 % _timestamp‚1.7844959159313974e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46256907 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚188 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82711866 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34862917 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46267064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚4599 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82701707 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24454758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3903573408256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360535614829056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360535614829056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271075412136960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356318327513615 _step‚377 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271075629391872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352537265347616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚356744144118272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46256907 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3791365004288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267172055983616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3791453671424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736ù‘Ñì™2Å- _wandbruntime‚3072  _runtime‚3072Žƒ)ÏÆ epoch‚11 0loss/FRt1/assignment_loss‚0.9406481017733971 2loss/FBt1/assignment_loss‚0.006258445416863253 .loss/FBt2/detection_loss‚0.566936369640891 trainer/global_step‚4599 % _timestamp‚1.7844959159323344e+09 _step‚378 1loss/FBt2/assignment_loss‚0.04319754931951208 /loss/FRt2/detection_loss‚0.5300175864926664 1loss/SRqqt1/detection_loss‚0.6772562925263957 /loss/FBt1/detection_loss‚0.5279040728399361 %loss/total_loss‚9.310473006792343 0loss/FRt2/assignment_loss‚2.1217628119050005 1loss/SRqqt1/assignment_loss‚0.635276538933442 2loss/SRqqt2/assignment_loss‚0.7235305127109988 /loss/FRt1/detection_loss‚1.5648307397673495 1loss/SRqqt2/detection_loss‚0.9728539854658905 _runtime‚3072.533884976õ;éƒ Ç† .loss/FBt2/detection_loss‚0.566936369640891 trainer/global_step‚4599 % _timestamp‚1.7844959159323344e+09 _step‚378 1loss/FBt2/assignment_loss‚0.04319754931951208 /loss/FRt2/detection_loss‚0.5300175864926664 1loss/SRqqt1/detection_loss‚0.6772562925263957 /loss/FBt1/detection_loss‚0.5279040728399361 %loss/total_loss‚9.310473006792343 0loss/FRt2/assignment_loss‚2.1217628119050005 1loss/SRqqt1/assignment_loss‚0.635276538933442 2loss/SRqqt2/assignment_loss‚0.7235305127109988 /loss/FRt1/detection_loss‚1.5648307397673495 1loss/SRqqt2/detection_loss‚0.9728539854658905 _runtime‚3072.533884976 epoch‚11 0loss/FRt1/assignment_loss‚0.9406481017733971 2loss/FBt1/assignment_loss‚0.006258445416863253úíí¦Ú8È‚(:­ ®þôÒÆêÙ¾ disk.rbd0.out‚0' disk.nvme0c0n1.out‚20391.7763671875 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 185.11328125* proc.memory.percent‚17.616844177246094 cpu‚25.801337895092384$ disk./.usageGB‚1171.595027923584 disk.dm-0.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚20391.7763671875# memory_percent‚98.8701581954956 proc.memory.rssMB‚ 2886.34375 network.recv‚839899# disk.nvme1n1.out‚9876.755859375 disk.md0.in‚0 disk.md0.out‚ 81.109375# disk.dm-0.out‚27690.39306640625" disk.dm-1.out‚2403.62255859375 disk.nvme0c0n1.in‚0 network.sent‚ 6.709933e+06( disk./.usagePercent‚76.3129801611193 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚9876.755859375 proc.cpu.threads‚17 disk.nvme1n1.in‚0Q·[®žÉ‚(:“ ®þôÒáø¹¿ gpu.0.gpu‚53.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.316' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.43866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Þ!BÊ‚(:· ½þôÒ¬´ý¾) disk./.usagePercent‚76.34508772444558% disk./.usageGB‚1172.0879592895508# disk.dm-0.out‚28576.62841796875 disk.rbd0.out‚0 proc.memory.rssMB‚ 2886.34375 network.recv‚844328 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚21067.77880859375& disk.nvme0n1.out‚21067.77880859375( disk.nvme1c1n1.out‚10091.65087890625 disk.md0.in‚0& proc.memory.availableMB‚ 185.109375" disk.dm-1.out‚2407.14892578125 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10091.65087890625$ memory_percent‚98.87018203735352* proc.memory.percent‚17.616844177246094 cpu‚25.80036005328346 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 81.64453125 network.sent‚ 6.773979e+06ˆyb™žË‚(:“ ½þôÒÕóí¿ gpu.0.gpu‚75.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚51.0 gpu.0.powerWatts‚231.048' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.01599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$†•2Ì- _wandbruntime‚3103  _runtime‚3103à=xÉÍà ' lr-AdamW/pg1‚0.00014255285977132147 ' lr-AdamW/pg2‚0.00014255285977132147 trainer/global_step‚4649 % _timestamp‚1.7844959465730486e+09 _runtime‚3103.174895932 _step‚379¤Œ†±ÎÎÈ % _timestamp‚1.7844959465730486e+09 _runtime‚3103.174895932 _step‚379 ' lr-AdamW/pg1‚0.00014255285977132147 ' lr-AdamW/pg2‚0.00014255285977132147 trainer/global_step‚4649ûPñæß2Ï- _wandbruntime‚3103  _runtime‚31036¶n’ÐÜO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46750082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3829929228800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269922710545920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚4649 _runtime‚3103.175688024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360426477767168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚193 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359997018293919 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364256512695296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24702506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273865922446336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3819413376309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36818623 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83568705 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35215645 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMoCFkPnitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7844959465739315e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364256406995968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943343441408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35215838 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10513143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚380 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3830017888768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46750082 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ö¤Ðç'ÑáO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚364256512695296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚273866148097024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269922804655616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46739939 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943211900416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3819500162735 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3829929228800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36818623 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46739939 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360426494806528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360426494806528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36818621 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚364256406995968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359996914468133 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36818621 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356177501091824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83558560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360426477767168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24702502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269922710545920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 trainer/global_step‚4649 _runtime‚3103.175688024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360426477767168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚193 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359997018293919 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364256512695296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24702506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚273865922446336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3819413376309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36818623 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83568705 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35215645 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7844959465739315e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364256406995968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943343441408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35215838 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10513143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚380 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3830017888768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46750082 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46750082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3829929228800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273923097837056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35222861 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3830714944512 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24707558 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46749759 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10515489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360501627433472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚364332342377984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83586168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36826252 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36826250 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3830714944512 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83576009 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3820283657447 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360501627433472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35223051 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3830803611648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10515303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3944155453440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364332448084480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3944021822464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360071988488125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36826250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3103.578397382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83586168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356251791617216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10515489 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360501627433472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚364332342377984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83586168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36826252 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36826250 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3830714944512 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83576009 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚186 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3820283657447 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360501627433472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35223051 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3830803611648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759916 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10515303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227740672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3944155453440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364332448084480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3944021822464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360071988488125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36826250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3103.578397382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83586168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356251791617216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273923097837056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35222861 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3830714944512 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ýT#2Õ- _wandbruntime‚3103  _runtime‚3103PƒœWÖ€ 2loss/SRqqt2/assignment_loss‚0.6093323937037664 1loss/FBt1/assignment_loss‚0.04373066350330201 1loss/SRqqt1/detection_loss‚0.7114308843406264 /loss/FRt1/detection_loss‚1.5322836562605695 1loss/SRqqt2/detection_loss‚0.9398169527529576 $loss/total_loss‚9.36718978494111 epoch‚11 % _timestamp‚1.784495946977°¦¡f¯3772e+09 _step‚382 0loss/FRt2/assignment_loss‚2.0724837417418596 1loss/FBt2/assignment_loss‚0.23213360303764727 /loss/FRt2/detection_loss‚0.5398379701077599 trainer/global_step‚4649 _runtime‚3103.579235664 0loss/FRt1/assignment_loss‚0.9236115738582824 1loss/SRqqt1/assignment_loss‚0.597897585412378 /loss/FBt1/detection_loss‚0.4779388910344922 /loss/FBt2/detection_loss‚0.6866918691874682v9ÎÀ ×… 2loss/SRqqt2/assignment_loss‚0.6093323937037664 1loss/FBt1/assignment_loss‚0.04373066350330201 1loss/SRqqt1/detection_loss‚0.7114308843406264 /loss/FRt1/detection_loss‚1.5322836562605695 1loss/SRqqt2/detection_loss‚0.9398169527529576 $loss/total_loss‚9.36718978494111 epoch‚11 % _timestamp‚1.7844959469773772e+09 _step‚382 0loss/FRt2/assignment_loss‚2.0724837417418596 1loss/FBt2/assignment_loss‚0.23213360303764727 /loss/FRt2/detection_loss‚0.5398379701077599 trainer/global_step‚4649 _runtime‚3103.579235664 0loss/FRt1/assignment_loss‚0.9236115738582824 1loss/SRqqt1/assignment_loss‚0.597897585412378 /loss/FBt1/detection_loss‚0.4779388910344922 /loss/FBt2/detection_loss‚0.6866918691874682þRóÐ@Ø‚(:µ ÌþôÒòÔŒ¿ network.sent‚ 6.789786e+06% disk.nvme1n1.out‚10095.3583984375# disk.dm-0.out‚28596.85791015625' disk.nvme0c0n1.out‚21085.5009765625 cpu‚25.799394371789642 proc.cpu.threads‚17 disk.dm-1.in‚0 network.recv‚847596% disk./.usageGB‚1172.1496047973633 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚10095.3583984375 disk.md0.out‚ 82.15234375 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.86727333068848' proc.memory.availableMB‚ 185.5859375" disk.dm-1.out‚2407.27001953125) disk./.usagePercent‚76.34910306447571 disk.md0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚21085.5009765625 proc.memory.rssMB‚ 2886.34375* proc.memory.percent‚17.616844177246094/Z$‹Ù‚(:’ ÌþôÒØõ¿¿ gpu.0.gpu‚92.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚116.04' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.68000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”)Í=Ú‚(:² ÛþôÒ­µƒ¿$ memory_percent‚98.86882305145264 network.sent‚ 6.839528e+06) disk./.usagePercent‚76.34909983431355& disk.nvme0n1.out‚21089.19091796875 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚10096.060546875* proc.memory.percent‚17.616844177246094 cpu‚25.798281707739477% disk./.usageGB‚1172.1495552062988 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2886.34375 proc.cpu.threads‚17 network.recv‚852009( disk.nvme0c0n1.out‚21089.19091796875 disk.md0.out‚82.40625# disk.dm-0.out‚28600.40869140625! disk.dm-1.out‚2407.5595703125( proc.memory.availableMB‚ 185.33203125& disk.nvme1c1n1.out‚10096.060546875 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0»k@`“Û‚(:ˆ ÛþôÒµîÁ¿ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.032' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I¸Ó"2Ü- _wandbruntime‚3134  _runtime‚3134ƒC„JÉÝà ' lr-AdamW/pg1‚0.00014214605693505994 ' lr-AdamW/pg2‚0.00014214605693505994 trainer/global_step‚4699 % _timestamp‚1.7844959779489353e+09 _runtime‚3134.552268552 _step‚383^÷¥ÎÞÈ ' lr-AdamW/pg2‚0.00014214605693505994 trainer/global_step‚4699 % _timestamp‚1.7844959779489353e+09 _runtime‚3134.552268552 _step‚383 ' lr-AdamW/pg1‚0.00014214605693505994ÿ=ÏJh2ß- _wandbruntime‚3134  _runtime‚3134‚drâ'àÜO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47232791 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47232791 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359892027361424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 % _timestamp‚1.7844959779495573e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364183961082368 trainer/global_step‚4699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3869367828992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3869367828992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37200073 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276713590902784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10620305 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3869279169024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368053240251392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10620480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364183961082368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3858651907117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368053345950720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37200073 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272729691055616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84432862 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368053345950720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚364183978121728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363750783094327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359892044400784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3983774597632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35575607 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84443007 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24955306 _runtime‚3134.552335582 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368053240251392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3869279169024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24955302 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276713371543552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37200071 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47242934 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35575786 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3983899847168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3858738693543 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272729596945920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363750679268541 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37200071 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0òÚŸ<ç'ááO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84443007 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24955306 _runtime‚3134.552335582 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368053240251392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3869279169024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24955302 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276713371543552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37200071 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47242934 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35575786 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3983899847168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3858738693543 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272729596945920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363750679268541 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37200071 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47232791 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47232791 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359892027361424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 % _timestamp‚1.7844959779495573e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364183961082368 trainer/global_step‚4699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3869367828992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3869367828992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37200073 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276713590902784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10620305 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3869279169024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368053240251392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10620480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364183961082368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3858651907117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368053345950720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37200073 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272729691055616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84432862 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368053345950720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚364183978121728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363750783094327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚175 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359892044400784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3983774597632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35575607 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682€yY·d2â- _wandbruntime‚3134  _runtime‚31343¼ÚAãæM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363825753288533 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚385 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24960362 % _timestamp‚1.7844959783508878e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47242611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24960358 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84460470 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚4699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3984586821632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37207702 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368129281339904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3859522188255 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚364259127788032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37207702 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3859435401717 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37207700 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368129281339904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3870153551872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 _runtime‚3134.952841691 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35583001 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47252768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3870153551872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37207700 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3984714161152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10622463 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚364259110748672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84450311 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35582821 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364259127788032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3870064884736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364259110748672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 O)á^«GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3870064884736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84450311 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47252768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368129175633408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276770321495552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272785734673920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359966334926176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359966317886816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84460470 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¡rjœñ&äëM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 _runtime‚3134.952841691 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35583001 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47252768 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3870153551872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37207700 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3984714161152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10622463 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚364259110748672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84450311 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35582821 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364259127788032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3870064884736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364259110748672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3870064884736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84450311 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47252768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368129175633408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276770321495552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272785734673920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359966334926176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359966317886816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84460470 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363825753288533 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚385 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24960362 % _timestamp‚1.7844959783508878e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47242611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24960358 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84460470 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚4699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3984586821632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37207702 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368129281339904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3859522188255 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚364259127788032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37207702 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3859435401717 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37207700 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368129281339904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3870153551872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898‚˜Ý~2å- _wandbruntime‚3134  _runtime‚31348ywâæ€ /loss/FBt1/detection_loss‚0.6835747414907191 trainer/global_step‚4699 1loss/SRqqt1/assignment_loss‚0.610186766285423 2loss/SRqqt2/assignment_loss‚0.5760472609364743 1loss/SRqqt1/detection_loss‚0.6162826786392813 /loss/FBt2/detection_loss‚0.6758775241069088 _runtime‚3134.953337282 _step‚386 0loss/FRt1/assignment_loss‚0.7472971007547665 /loss/FRt2/assignment_loss‚2.111600799362907 1loss/FBt2/assignment_loss‚0.02694523512576277 /loss/FRt1/detection_loss‚1.4278339419959616 /loss/FRt2/detection_loss‚0.5289436342213275 1loss/SRqqt2/detection_loss‚1.1292526137008345 epoch‚11 % _timestamp‚1.7844959783518262e+09 2loss/FBt1/assignment_loss‚0.026336659549254904 $loss/total_loss‚9.16017895616962+£]ï ç… /loss/FBt1/detection_loss‚0.6835747414907191 trainer/global_step‚4699 1loss/SRqqt1/assignment_loss‚0.610186766285423 2loss/SRqqt2/assignment_loss‚0.5760472609364743 1loss/SRqqt1/detection_loss‚0.6162826786392813 /loss/FBt2/detection_loss‚0.6758775241069088 _runtime‚3134.953337282 _step‚386 0loss/FRt1/assignment_loss‚0.7472971007547665 /loss/FRt2/assignment_loss‚2.111600799362907 1loss/FBt2/assignment_loss‚0.02694523512576277 /loss/FRt1/detection_loss‚1.4278339419959616 /loss/FRt2/detection_loss‚0.5289436342213275 1loss/SRqqt2/detection_loss‚1.1292526137008345 epoch‚11 % _timestamp‚1.7844959783518262e+09 2loss/FBt1/assignment_loss‚0.026336659549254904 $loss/total_loss‚9.16017895616962‚$iT;è‚(:° êþôÒ’¨¡¿ disk.rbd0.out‚0( disk.nvme0c0n1.out‚21090.72119140625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚10096.7255859375& proc.memory.availableMB‚ 185.328125 proc.cpu.threads‚17 network.sent‚ 6.85823e+06) disk./.usagePercent‚76.34916543145266 disk.md0.in‚0! disk.dm-1.out‚2407.6494140625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 proc.memory.rssMB‚ 2886.34375 cpu‚25.797253897971956 network.recv‚855081 disk.md0.out‚82.84375 disk.dm-0.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚21090.72119140625 disk.nvme1n1.in‚0$ memory_percent‚98.86884689331055* proc.memory.percent‚17.616844177246094$ disk./.usageGB‚1172.150562286377# disk.dm-0.out‚28601.56689453125 disk.dm-1.in‚0% disk.nvme1n1.out‚10096.7255859375<‘žé‚(:“ êþôÒëâÕ¿ gpu.0.gpu‚92.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚226.006' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.33533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»Ä<ê‚(:± ùþôÒ’Îç¾ cpu‚25.79632632142597% disk./.usageGB‚1172.1505126953125& disk.nvme0n1.out‚21091.70166015625& disk.nvme1c1n1.out‚10097.232421875( proc.memory.availableMB‚ 185.54296875* proc.memory.percent‚17.616844177246094 proc.cpu.threads‚17 network.recv‚859577 disk.dm-1.out‚2407.818359375( disk.nvme0c0n1.out‚21091.70166015625 disk.nvme1n1.in‚0 disk.dm-0.in‚0) disk./.usagePercent‚76.34916220129051 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚10097.232421875 disk.md0.in‚0 disk.md0.out‚ 83.078125 network.sent‚ 6.908459e+06 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0# disk.dm-0.out‚28602.37353515625$ memory_percent‚98.86753559112549 proc.memory.rssMB‚ 2886.34375#=@“ë‚(:ˆ ùþôÒçö˜¿ gpu.0.gpu‚98.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.227' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.409 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iº:ì‚(:¯ ˆÿôÒ᜕¿& disk.nvme1n1.out‚10098.18310546875 disk.dm-0.in‚0# disk.dm-0.out‚28605.18408203125 disk.rbd0.out‚0 network.recv‚862938 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚21094.708984375 disk.nvme0n1.in‚0# memory_percent‚98.8675594329834 proc.cpu.threads‚17 disk.dm-1.out‚2407.908203125) disk./.usagePercent‚76.34917338262103 disk.md0.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚21094.708984375 proc.memory.rssMB‚ 2886.34375* proc.memory.percent‚17.616844177246094 cpu‚25.79531519659179( disk.nvme1c1n1.out‚10098.18310546875 disk.md0.out‚ 83.578125' proc.memory.availableMB‚ 185.5390625 disk.dm-1.in‚0 network.sent‚ 6.913365e+06% disk./.usageGB‚1172.1506843566895 disk.nvme1n1.in‚0朰~í‚(:’ ˆÿôÒÈÚ¿ gpu.0.gpu‚21.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.472' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.15733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xú—w2î- _wandbruntime‚3165  _runtime‚3165ß[PTÉïà ' lr-AdamW/pg1‚0.00014173527703266716 ' lr-AdamW/pg2‚0.00014173527703266716 trainer/global_step‚4749 % _timestamp‚1.7844960090242312e+09 _runtime‚3165.625996874 _step‚387<¿yÃÎðÈ % _timestamp‚1.7844960090242312e+09 _runtime‚3165.625996874 _step‚387 ' lr-AdamW/pg1‚0.00014173527703266716 ' lr-AdamW/pg2‚0.00014173527703266716 trainer/global_step‚4749ƒÀŠ¥T2ñ- _wandbruntime‚3165  _runtime‚3165B¶–Aâ'òÜO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37581521 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47735786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25208106 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371850179206144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367504547894735 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚371850073506816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85317309 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 % _timestamp‚1.7844960090248172e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35935130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3897890437925 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚363606553631024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279560807112192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3908629109248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚367941461436928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47725643 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35934950 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371850073506816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47735786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4024451112960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4024323766272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3908717769216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37581523 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279°7¸NóáO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35934950 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371850073506816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47735786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4024451112960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4024323766272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3908717769216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37581523 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37581521 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47735786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25208106 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚371850179206144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367504547894735 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚371850073506816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85317309 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 % _timestamp‚1.7844960090248172e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35935130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3897890437925 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚363606553631024 JBDeviceStatsMot«¦Ì™nitor.on_train_batch_start/segment.small_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279560807112192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3908629109248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚367941461436928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47725643 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35942152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 _step‚389 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279617754246144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3898673932525 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371926114595328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10728994 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37589152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368016594063872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35942321 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367579518088941 trainer/global_step‚4749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85334772 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚363680844156416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3909414824960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279617971501056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47745620 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85324613 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37589150 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3909503492096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3909503492096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85324613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3909414824960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25213162 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275592621073920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275592715183616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚363680861195776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367579621914839 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37589150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275592715183616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚363680861195776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367579621914839 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37589150 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35942152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 _step‚389 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279617754246144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3898673932525 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371926114595328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10728994 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37589152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368016594063872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35942321 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367579518088941 trainer/global_step‚4749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85334772 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚363680844156416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3909414824960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279617971501056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47745620 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85324613 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37589150 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3909503492096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3909503492096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85324613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3909414824960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25213162 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840…Xx7÷2÷- _wandbruntime‚3166  _runtime‚3166Cï½øû 2loss/SRqqt1/assignment_loss‚0.5791918061356585 /loss/FBt2/assignment_loss‚0.090529530464394 0loss/SRqqt2/detection_loss‚1.099431030983912 /loss/FBt1/detection_loss‚0.5558189014151367 /loss/FBt2/detection_loss‚0.6746052057781028 %loss/total_loss‚9.243857417036999 1loss/SRqqt1/detection_loss‚0.6418047098911241 epoch‚11 trainer/global_step‚4749 $ _timestamp‚1.784496009427707e+09 _step‚390 /loss/FRt2/assignment_loss‚2.069948280606293 2loss/SRqqt2/assignment_loss‚0.5661013028254425 .loss/FRt2/detection_loss‚0.551511717075946 _runtime‚3166.029278092 0loss/FRt1/assignment_loss‚0.8262920529992731 0loss/FBt1/assignment_loss‚0.0403543151120449 /loss/FRt1/detection_loss‚1.5482685637496714å'íù€ _step‚390 /loss/FRt2/assignment_loss‚2.069948280606293 2loss/SRqqt2/assignment_loss‚0.5661013028254425 .loss/FRt2/detection_loss‚0.551511717075946 _runtime‚3166.029278092 0loss/FRt1/assignment_loss‚0.8262920529992731 0loss/FBt1/assignment_loss‚0.0403543151120449 /loss/FRt1/detection_loss‚1.5482685637496714 2loss/SRqqt1/assignment_loss‚0.5791918061356585 /loss/FBt2/assignment_loss‚0.090529530464394 0loss/SRqqt2/detection_loss‚1.099431030983912 /loss/FBt1/detection_loss‚0.5558189014151367 /loss/FBt2/detection_loss‚0.6746052057781028 %loss/total_loss‚9.243857417036999 1loss/SRqqt1/detection_loss‚0.6418047098911241 epoch‚11 trainer/global_step‚4749 $ _timestamp‚1.784496009427707e+09†?Y;ú‚(:° —ÿôÒ‘Ѧ¿ disk.md0.in‚0 disk.dm-0.in‚0! disk.dm-1.out‚2408.0107421875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚10098.8505859375 disk.nvme1n1.in‚0 disk.md0.out‚ 83.9609375 cpu‚25.79439561390998 network.sent‚ 6.928213e+06) disk./.usagePercent‚76.34916468603062# disk.dm-0.out‚28605.80908203125& proc.memory.availableMB‚ 193.578125* proc.memory.percent‚17.723846435546875 proc.cpu.threads‚17% disk.nvme1n1.out‚10098.8505859375 disk.dm-1.in‚0# memory_percent‚98.8184928894043 proc.memory.rssMB‚2903.875 network.recv‚865977% disk./.usageGB‚1172.1505508422852 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21095.57861328125& disk.nvme0n1.out‚21095.57861328125I×–žû‚(:“ —ÿôÒïø׿ gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚119.083' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.69433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉsU…žü‚(:“ ¦ÿôҚ؈¿ gpu.0.gpu‚68.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.14' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.379999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V3xAý‚(:¶ ¦ÿôÒ°âØ¿( disk.nvme0c0n1.out‚21097.27490234375& disk.nvme1n1.out‚10099.60400390625# memory_percent‚98.8184928894043& proc.memory.availableMB‚ 193.578125 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0* proc.memory.percent‚17.809057235717773 proc.cpu.threads‚17 network.sent‚ 6.979458e+06 network.recv‚870018% disk./.usageGB‚1171.9129676818848 disk.rbd0.out‚0 disk.md0.out‚ 84.50390625! disk.dm-1.out‚2408.0654296875 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚21097.27490234375( disk.nvme1c1n1.out‚10099.60400390625 disk.md0.in‚0" proc.memory.rssMB‚ 2917.8359375 cpu‚25.793324508384334) disk./.usagePercent‚76.33368947610387 disk.dm-0.in‚0! disk.dm-0.out‚28607.048828125 disk.rbd0.in‚0d¢e2þ- _wandbruntime‚3197  _runtime‚3197þ5ÇÿÁ _runtime‚3197.003919446 _step‚391 & lr-AdamW/pg1‚0.0001413205464477687 & lr-AdamW/pg2‚0.0001413205464477687 trainer/global_step‚4799 % _timestamp‚1.7844960404020474e+09SÍI€Ì€Æ % _timestamp‚1.7844960404020474e+09 _runtime‚3197.003919446 _step‚391 & lr-AdamW/pg1‚0.0001413205464477687 & lr-AdamW/pg2‚0.0001413205464477687 trainer/global_step‚4799‡¬û„û2- _wandbruntime‚3197  _runtime‚3197n-ŒÆ ‚ÜO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282408188451328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 % _timestamp‚1.7844960404030554e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚375647012461568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3947979049472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25460902 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278343369745920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37962971 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10833018 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86191611 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3948067709440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86181466 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48218495 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3947979049472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37962973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48218495 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37962971 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375646906762240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚371698944752128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚371698927712768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371698944752128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3948067709440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚07x<¨ç'ƒáO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10833018 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86191611 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3948067709440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86181466 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48218495 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3947979049472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37962973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48218495 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37962971 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚375646906762240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚371698944752128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚371698927712768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371698944752128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3948067709440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282408188451328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 % _timestamp‚1.7844960404030554e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚375647012461568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3947979049472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25460902 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219359232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚178 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278343369745920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37962971 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367395370426016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚375722842144256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48238472 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36300894 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375722947850752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚371774077379072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3948853432320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10834936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 % _timestamp‚1.7844960408027692e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282465359393792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚4799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚371774077379072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚393 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367395387465376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3937912463333 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86198915 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3948764765184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3948764765184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86198915 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278399507473920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37970602 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3948853432320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25465958 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37970600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4065757810176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86209074 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚282465135847424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0§cFNù†êM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚371774094418432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚375722842144256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48238472 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36300894 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375722947850752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚371774077379072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3948853432320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10834936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_¢£kô÷batch_end/requested_bytes.small_pool.current‚86786538 % _timestamp‚1.7844960408027692e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282465359393792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 trainer/global_step‚4799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚371774077379072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 _step‚393 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367395387465376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3937912463333 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86198915 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3948764765184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3948764765184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86198915 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278399507473920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37970602 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3948853432320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25465958 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37970600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4065757810176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86209074 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚282465135847424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367395370426016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E| =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7309390306472778 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9911620020866394 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925859570503235 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2338908463716507 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.291131854057312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8560742139816284 particle/accuracy_1_of_2‚1 +particle/sensitivity‚0.6315572261810303 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9709802269935608 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7782652974128723 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8173091411590576 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08466005325317383 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4791666865348816 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9849055409431458 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46031415462493896 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921150207519531 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6438397169113159 jet/accuracy_3_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2527148425579071 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5904180407524109 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.896215558052063 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6561230421066284 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8192553520202637 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05309824272990227 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3000394403934479 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937974810600281 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28495630621910095 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9828736186027527 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6132577061653137 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10234326124191284 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29976245760917664 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6778151988983154 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950113892555237 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7128245234489441 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7466012835502625 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9849206805229187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6873807311058044 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20416665077209473 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976010322570801 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7884615659713745 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚8.957645933012728 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7175465226173401 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3055555522441864 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.753743052482605 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_1_of_4‚0.9687806963920593 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6964011192321777 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24215984344482422 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5474085211753845 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7591689229011536 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_2‚0.15687862038612366 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5980830192565918 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8560742139816284 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6469191908836365 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.641974151134491 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_3_of_4‚0.09040666371583939 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3273408114910126 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6707503199577332 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6756696105003357 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9643781781196594 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21820034086704254 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.615969181060791 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3048216700553894 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05233974754810333 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28914445638656616 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.619009256362915 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 epoch‚11 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2669256627559662 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901041269302368 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28222185373306274 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8686555027961731 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40317225456237793 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5829064846038818 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7027400135993958 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910026788711548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_2_of_4‚0.8263686895370483 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2501467764377594 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6772006154060364 6validation_average_jet_accuracy‚0.4005056619644165 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965295195579529 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3445007801055908 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7079717516899109 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13854378461837769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45946216583251953 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.737808108329773 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8254913091659546 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12523499131202698 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.451292484998703 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05002817139029503 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_2_of_3‚0.5707167387008667 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 trainer/global_step‚4799 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6772006154060364 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚3210.774666811 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7524325847625732 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9912415742874146 +particle/specificity‚0.9122297167778015 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21261298656463623 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23013140261173248 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.042104609310626984 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8940092921257019 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9091368317604065 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3055555522441864 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.821617603302002 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6732919812202454 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46527454257011414 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9749067425727844 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8143407106399536 /particle/accuracy_4_of_4‚0.9530941247940063 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22858861088752747 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30111297965049744 particle/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6731469631195068 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007558579673059285 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04869222640991211 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.475565642118454 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.043992262333631516 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34953704476356506 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36046645045280457 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887978434562683 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.214436873793602 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.042104609310626984 *jet/accuracy_1_of_2‚0.8084503412246704 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_3‚0.9320325255393982 _step‚395 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9938266277313232 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6838481426239014 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 )jet/accuracy_1_of_1‚0.364982932806015 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1542322337627411 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966605305671692 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9821890592575073 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.049549393355846405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14064446091651917 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8121665716171265 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7755284905433655 +jet/accuracy_3_of_3‚0.07120610773563385 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47911810874938965 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23906299471855164 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5449806451797485 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4009370803833008 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9958572387695312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22009137272834778 particle/accuracy_2_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.048418670892715454 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7563204765319824 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0007558579673059285 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8390461802482605 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6575967669487 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.291131854057312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6187192797660828 /particle/accuracy_3_of_3‚0.9994933605194092 (particle/accuracy‚0.8519576787948608 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9770429730415344 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 particle/accuracy_4_of_5‚1 % _timestamp‚1.7844960541724095e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20864422619342804 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966605305671692 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04109245911240578 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7655024528503418 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7119496464729309 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7581325769424438 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472222089767456 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_2‚0.15687862038612366 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5980830192565918 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8560742139816284 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6469191908836365 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.641974151134491 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9972512125968933 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_3_of_4‚0.09040666371583939 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3273408114910126 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6707503199577332 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6756696105003357 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9643781781196594 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21820034086704254 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.615969181060791 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3048216700553894 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05233974754810333 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28914445638656616 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.619009256362915 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 epoch‚11 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2669256627559662 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901041269302368 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28222185373306274 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8686555027961731 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40317225456237793 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5829064846038818 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7027400135993958 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910026788711548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_2_of_4‚0.8263686895370483 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2501467764377594 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6772006154060364 6validation_average_jet_accuracy‚0.4005056619644165 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965295195579529 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3445007801055908 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7079717516899109 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13854378461837769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45946216583251953 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.737808108329773 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8254913091659546 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12523499131202698 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.451292484998703 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05002817139029503 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_2_of_3‚0.5707167387008667 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 trainer/global_step‚4799 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6772006154060364 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚3210.774666811 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7524325847625732 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_4‚0.9912415742874146 +particle/y2ê–Áspecificity‚0.9122297167778015 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21261298656463623 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23013140261173248 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.042104609310626984 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8940092921257019 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9091368317604065 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3055555522441864 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.821617603302002 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6732919812202454 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46527454257011414 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9749067425727844 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8143407106399536 /particle/accuracy_4_of_4‚0.9530941247940063 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22858861088752747 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30111297965049744 particle/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6731469631195068 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007558579673059285 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04869222640991211 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.475565642118454 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.043992262333631516 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34953704476356506 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36046645045280457 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887978434562683 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.214436873793602 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.042104609310626984 *jet/accuracy_1_of_2‚0.8084503412246704 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_3‚0.9320325255393982 _step‚395 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9938266277313232 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6838481426239014 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 )jet/accuracy_1_of_1‚0.364982932806015 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1542322337627411 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966605305671692 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9821890592575073 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.049549393355846405 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14064446091651917 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8121665716171265 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7755284905433655 +jet/accuracy_3_of_3‚0.07120610773563385 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47911810874938965 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23906299471855164 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5449806451797485 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4009370803833008 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9958572387695312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22009137272834778 particle/accuracy_2_of_4‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.048418670892715454 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7563204765319824 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0007558579673059285 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8390461802482605 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6575967669487 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.291131854057312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6187192797660828 /particle/accuracy_3_of_3‚0.9994933605194092 (particle/accuracy‚0.8519576787948608 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9770429730415344 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 particle/accuracy_4_of_5‚1 % _timestamp‚1.7844960541724095e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20864422619342804 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966605305671692 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04109245911240578 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7655024528503418 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7119496464729309 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7581325769424438 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472222089767456 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7309390306472778 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9911620020866394 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925859570503235 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2338908463716507 ("Purity/2FRt1SRqqt0FBt/event_purity‚0‹€H°F‚(:» ÄÿôÒµÑؾ network.sent‚ 7.044703e+06$ disk./.usageGB‚1171.913387298584 disk.nvme0n1.in‚0( disk.nvme0c0n1.out‚21107.60302734375( proc.memory.availableMB‚ 194.96484375 proc.cpu.threads‚17 disk.dm-0.in‚0# disk.dm-0.out‚28616.92626953125) disk./.usagePercent‚76.33371680824517& disk.nvme0n1.out‚21107.60302734375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10164.89208984375 disk.md0.out‚ 85.27734375 network.recv‚877472 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ memory_percent‚98.81002902984619) proc.memory.percent‚18.07692050933838 cpu‚25.803746071290234( disk.nvme1c1n1.out‚10164.89208984375 disk.md0.in‚0" disk.dm-1.out‚2472.14501953125# proc.memory.rssMB‚ 2961.72265625fØ[óœ‚(:‘ ÄÿôÒ–ßµ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚221.963' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.98766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KØ؆G‘‚(:¼ ÓÿôÒ®ú¾$ memory_percent‚98.81002902984619 cpu‚25.802648791287723 proc.cpu.threads‚17 network.sent‚ 7.103685e+06# disk.dm-0.out‚28617.73095703125" disk.dm-1.out‚2536.12548828125 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 194.96484375) disk./.usagePercent‚76.33373395295199 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚21108.44677734375 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10229.59130859375) proc.memory.percent‚18.07692050933838% disk./.usageGB‚1171.9136505126953 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚21108.44677734375# proc.memory.rssMB‚ 2961.72265625 network.recv‚882097 disk.md0.out‚ 85.62890625( disk.nvme1c1n1.out‚10229.59130859375%nºž’‚(:“ ÓÿôÒðά¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.093' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.03099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!¸[Û2“- _wandbruntime‚3242  _runtime‚3242–gùȔ _runtime‚3242.195701189 _step‚396 ' lr-AdamW/pg1‚0.00014090189181773497 ' lr-AdamW/pg2‚0.00014090189181773497 trainer/global_step‚4849 $ _timestamp‚1.784496085593872e+09ŽÓÈÆÍ•Ç ' lr-AdamW/pg1‚0.00014090189181773497 ' lr-AdamW/pg2‚0.00014090189181773497 trainer/global_step‚4849 $ _timestamp‚1.784496085593872e+09 _runtime‚3242.195701189 _step‚396Œ½!ð2–- _wandbruntime‚3242  _runtime‚3242gÿE©á'—ÛO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87260075 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372009073206416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38466223 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38466221 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36809142 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286074799055360 _runtime‚3242.199321715 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380455762390016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376452915363328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376452915363328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281953621350912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4120960442368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3991782758107 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _step‚397 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376000959790309 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380455656690688 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4002847026688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38466223 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376000855964523 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48803997 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10987185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4002758366720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25821957 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48793854 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚376452898323968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376452898323968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36809329 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 $ _timestamp‚1.784496085594737e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380455656690688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286074581793280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380455762390016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48803997 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281953715460608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380455656690688 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4002847026688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38466223 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚376000855964523 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48803997 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10987185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4002758366720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25821957 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48793854 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚376452898323968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376452898323968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36809329 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 $ _timestamp‚1.784496085594737e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380455656690688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286074581793280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217262080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380455762390016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48803997 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281953715460608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36809142 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286074799055360 _runtime‚3242.199321715 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380455762390016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376452915363328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376452915363328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281953621350912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4120960442368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3991782758107 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _step‚397 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376000959790309 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200X}¿2™- _wandbruntime‚3242  _runtime‚3242² ³+šæM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚193 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376076033810413 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87277524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚189 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48813831 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380531592072704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376528065029632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376528047990272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4121769389568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25827017 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStañ¾ÁtsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25827013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38473850 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4121894631936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376528065029632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38473852 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282009759078912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4003632749568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844960859959512e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48813831 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10989486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36816310 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286131747820544 trainer/global_step‚4849 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376075929984515 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282009853188608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380531697779200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286131528468480 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87277524 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87287683 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48803674 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36816503 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130ØØ™ñ&›ëM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286131747820544 trainer/global_step‚4849 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376075929984515 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282009853188608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380531697779200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286131528468480 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87277524 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87287683 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48803674 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36816503 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚193 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376076033810413 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87277524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚189 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48813831 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380531592072704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376528065029632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376528047990272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4121769389568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25827017 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25827013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38473850 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4121894631936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376528065029632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38473852 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282009759078912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4003632749568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844960859959512e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48813831 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10989486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36816310ŽÄÖ`2œ- _wandbruntime‚3242  _runtime‚3242WkÃ(€ epoch‚12 /loss/FRt2/assignment_loss‚2.025098622925692 2loss/SRqqt1/assignment_loss‚0.6188594915553104 1loss/SRqqt1/detection_loss‚0.5972424464542097 trainer/global_step‚4849 2loss/FBt1/assignment_loss‚0.004801039100702774 1loss/FBt2/assignment_loss‚0.21140777603542346 -loss/FBt2/detection_loss‚0.62758662274573 _runtime‚3242.598433627 0loss/FRt1/assignment_loss‚0.8761276945887334 2loss/SRqqt2/assignment_loss‚0.6526826735415097 /loss/FRt1/detection_loss‚1.5071550590372031 1loss/SRqqt2/detection_loss‚1.0532929049214532 %loss/total_loss‚9.382582656649385 % _timestamp‚1.7844960859968865e+09 _step‚399 /loss/FRt2/detection_loss‚0.5676097078960627 /loss/FBt1/detection_loss‚0.6407186178473545Ÿ4`& ž… _runtime‚3242.598433627 0loss/FRt1/assignment_loss‚0.8761276945887334 2loss/SRqqt2/assignment_loss‚0.6526826735415097 /loss/FRt1/detection_loss‚1.5071550590372031 1loss/SRqqt2/detection_loss‚1.0532929049214532 %loss/total_loss‚9.382582656649385 % _timestamp‚1.7844960859968865e+09 _step‚399 /loss/FRt2/detection_loss‚0.5676097078960627 /loss/FBt1/detection_loss‚0.6407186178473545 epoch‚12 /loss/FRt2/assignment_loss‚2.025098622925692 2loss/SRqqt1/assignment_loss‚0.6188594915553104 1loss/SRqqt1/detection_loss‚0.5972424464542097 trainer/global_step‚4849 2loss/FBt1/assignment_loss‚0.004801039100702774 1loss/FBt2/assignment_loss‚0.21140777603542346 -loss/FBt2/detection_loss‚0.62758662274573T"@Ÿ‚(:µ âÿôÒ约¿ network.recv‚885349) disk./.usagePercent‚76.33373494684803 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚21117.556640625 disk.nvme1n1.in‚0 disk.md0.out‚ 86.1796875# proc.memory.rssMB‚ 2961.72265625 disk.dm-0.in‚0" disk.dm-1.out‚2536.24658203125 disk.rbd0.in‚0$ disk.nvme0n1.out‚21117.556640625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10232.26123046875) proc.memory.percent‚18.07692050933838 cpu‚25.803319225183643 network.sent‚ 7.119593e+06% disk./.usageGB‚1171.9136657714844 proc.cpu.threads‚17" disk.dm-0.out‚28628.2216796875 disk.dm-1.in‚0 disk.nvme0n1.in‚0& disk.nvme1n1.out‚10232.26123046875 disk.md0.in‚0$ memory_percent‚98.81176948547363' proc.memory.availableMB‚ 194.6796875ØÁÜž ‚(:“ âÿôÒóŽÒ gpu.0.gpu‚56.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.127' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.04233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:lq[<¡‚(:± ñÿôÒžÿ¾# disk./.usageGB‚1172.15189743042 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 cpu‚25.806049780855922 network.recv‚889639 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚21218.466796875( disk.nvme1c1n1.out‚10271.02490234375) proc.memory.percent‚18.07692050933838& disk.nvme1n1.out‚10271.02490234375 disk.md0.out‚86.53125" disk.dm-0.out‚28763.7646484375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0# memory_percent‚98.8102674484253 proc.cpu.threads‚17) disk./.usagePercent‚76.34925239735678 disk.md0.in‚0 disk.dm-0.in‚0" disk.dm-1.out‚2539.61962890625& disk.nvme0c0n1.out‚21218.466796875( proc.memory.availableMB‚ 194.92578125# proc.memory.rssMB‚ 2961.72265625 network.sent‚ 7.170942e+06293Ο¢‚(:” ñÿôÒÅÿ«¿ gpu.0.gpu‚98.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚124.614' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.538000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'3™”2£- _wandbruntime‚3274  _runtime‚3274¾?“õɤà % _timestamp‚1.7844961180289464e+09 _runtime‚3274.630785193 _step‚400 ' lr-AdamW/pg1‚0.00014047934003197062 ' lr-AdamW/pg2‚0.00014047934003197062 trainer/global_step‚4899¬çñäÎ¥È ' lr-AdamW/pg2‚0.00014047934003197062 trainer/global_step‚4899 % _timestamp‚1.7844961180289464e+09 _runtime‚3274.630785193 _step‚400 ' lr-AdamW/pg1‚0.00014047934003197062»ª2K2¦- _wandbruntime‚3274  _runtime‚3274óêáâ'§ÜO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4031108075341 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380210381639168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38847673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384252595645440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4031021288915 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 _step‚401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380210398678528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49296849 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37170743 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384252489946112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88134377 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380210398678528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379754620764931 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4042108306944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26074757 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚3274.631956355 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49286706 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4161697495552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚375723616515376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384252489946112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4042196966912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11095798 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288922075802624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11095982 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88144522 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37170555 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚375723599476016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49296849 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49286706 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760‘™£ ¨áO G>DeÛé²Û'viceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384252595645440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4031021288915 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 _step‚401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380210398678528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49296849 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37170743 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384252489946112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88134377 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380210398678528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379754620764931 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4042108306944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26074757 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚3274.631956355 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49286706 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4161697495552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚375723616515376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384252489946112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4042196966912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11095798 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288922075802624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11095982 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88144522 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37170555 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚375723599476016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49296849 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49286706 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4031108075341 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380210381639168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38847673‘Ž ßâ2©- _wandbruntime‚3275  _runtime‚3275é)½ ì&ªæM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4042982689792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380285548344832 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4042894022656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380285531305472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38855300 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚375797907040768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49296526 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288979244713472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288979019069952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379829694784923 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4031891570053 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4031804783515 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _step‚402 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49306683 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38855302 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88161985 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38855302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384328531034624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11097902 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284816739588608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4162505124864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384328531034624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380285548344832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚375797890001408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379829798610821 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88151826 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37177909 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284816645478912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380285531305472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4042982689792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚190 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4042894022656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚4899 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88161985 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49306683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3275.049823335 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37177715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26079817 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¸N^9ñ&«ëM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88161985 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49306683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3275.049823335 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37177715 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26079817 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4042982689792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380285548344832 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4042894022656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚380285531305472 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38855300 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚375797907040768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49296526 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288979244713472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288979019069952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379829694784923 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4031891570053 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4031804783515 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _step‚402 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49306683 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38855302 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88161985 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38855302 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384328531034624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11097902 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284816739588608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4162505124864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384328531034624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380285548344832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚375797890001408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379829798610821 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88151826 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37177909 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284816645478912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380285531305472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4042982689792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚190 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4042894022656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 trainer/global_step‚4899’”t=2¬- _wandbruntime‚3275  _runtime‚3275K`9Ÿ­‚ /loss/FRt1/assignment_loss‚0.921444790630505 2loss/SRqqt1/assignment_loss‚0.6157435311746928 2loss/SRqqt2/assignment_loss‚0.7826468505906651 2loss/FBt1/assignment_loss‚0.027080890222525734 1loss/FBt2/assignment_loss‚0.11349658041771053 %loss/total_loss‚9.692339434404147 /loss/FRt2/detection_loss‚0.5665038126669877 1loss/SRqqt1/detection_loss‚0.6532042738617445 1loss/SRqqt2/detection_loss‚0.9839341916353596 /loss/FBt1/detection_loss‚0.4720291038708383 /loss/FBt2/detection_loss‚1.0268734445045185 % _timestamp‚1.7844961184488072e+09 epoch‚12 trainer/global_step‚4899 _step‚403 0loss/FRt2/assignment_loss‚1.9978899290719008 /loss/FRt1/detection_loss‚1.5314920357566986 _runtime‚3275.054278083vkX ®‡ /loss/FBt2/detection_loss‚1.0268734445045185 % _timestamp‚1.7844961184488072e+09 epoch‚12 trainer/global_step‚4899 _step‚403 0loss/FRt2/assignment_loss‚1.9978899290719008 /loss/FRt1/detection_loss‚1.5314920357566986 _runtime‚3275.054278083 /loss/FRt1/assignment_loss‚0.921444790630505 2loss/SRqqt1/assignment_loss‚0.6157435311746928 2loss/SRqqt2/assignment_loss‚0.7826468505906651 2loss/FBt1/assignment_loss‚0.027080890222525734 1loss/FBt2/assignment_loss‚0.11349658041771053 %loss/total_loss‚9.692339434404147 /loss/FRt2/detection_loss‚0.5665038126669877 1loss/SRqqt1/detection_loss‚0.6532042738617445 1loss/SRqqt2/detection_loss‚0.9839341916353596 /loss/FBt1/detection_loss‚0.4720291038708383“h&¢âB¯‚(:· €€õÒÒõ€¿# memory_percent‚98.8105297088623# proc.memory.rssMB‚ 2961.72265625 cpu‚25.80844707843577 proc.cpu.threads‚17 network.recv‚893045 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10299.26416015625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10299.26416015625( disk.nvme0c0n1.out‚21303.97412109375 network.sent‚ 7.186947e+06# disk./.usageGB‚1172.15189743042 disk.dm-0.in‚0" disk.dm-1.out‚2539.79150390625 disk.nvme0n1.in‚0& disk.nvme0n1.out‚21303.97412109375' proc.memory.availableMB‚ 194.8828125 disk.md0.out‚ 86.82421875) proc.memory.percent‚18.07692050933838) disk./.usagePercent‚76.34925239735678# disk.dm-0.out‚28876.68896484375 disk.rbd0.in‚0 disk.rbd0.out‚0 j: N°‚(:’ €€õÒ¿•¯¿ gpu.0.gpu‚52.0 gpu.0.memory‚9, gpu.0.memPãû°OoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚223.003' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.33433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b¾rA±‚(:¶ €õÒ”çÚ¾ network.sent‚ 7.236736e+06) disk./.usagePercent‚76.34925090651271% disk./.usageGB‚1172.1518745422363 disk.nvme0c0n1.in‚0% disk.nvme1n1.out‚10299.9189453125 network.recv‚897440 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚28877.21240234375# memory_percent‚98.8107442855835( proc.memory.availableMB‚ 194.84765625' disk.nvme1c1n1.out‚10299.9189453125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21304.63232421875& disk.nvme0n1.out‚21304.63232421875 cpu‚25.81075236199074 proc.cpu.threads‚17! disk.dm-1.out‚2540.1083984375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 87.0390625 disk.dm-0.in‚0# proc.memory.rssMB‚ 2961.72265625) proc.memory.percent‚18.07692050933838—)Óž²‚(:“ €õÒð…œ¿ gpu.0.gpu‚32.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.848' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.282666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò€êæ2³- _wandbruntime‚3308  _runtime‚3308EÆ›ïÈ´Â ' lr-AdamW/pg1‚0.00014005291823018716 ' lr-AdamW/pg2‚0.00014005291823018716 trainer/global_step‚4949 $ _timestamp‚1.784496151404014e+09 _runtime‚3308.005819594 _step‚404¹vg굂 trainer/global_step‚4949 $ _timestamp‚1.784496151404014e+09 _runtime‚3308.005819594 _step‚404 ' lr-AdamW/pg1‚0.00014005291823018716 ' lr-AdamW/pg2‚0.00014005291823018716”nA92¶- _wandbruntime‚3308  _runtime‚3308çzûÐá'·ÛO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39229123 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 _step‚405 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37531129 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4202105575936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4081458247168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383508385565339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383967881993728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291769714891776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39229123 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚379438125745616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39229121 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4070259819723 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26327561 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388049428900864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388049323201536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26327557 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49779558 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383967864954368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11203572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287567394150912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388049323201536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4081546907136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383508489391125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49789701 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37531302 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291769499726848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89008679 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39229121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49779558 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383967864954368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11203572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287567394150912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388049323201536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4081546907136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383508489391125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49789701 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37531302 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291769499726848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89008679 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39229121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚173 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383508385565339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383967881993728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291769714891776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39229123 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚169 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚379438125745616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39229121 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4070259819723 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26327561 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388049428900864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388049323201536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26327557 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291826443256320 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89026128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39236750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4202911377408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11205793 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4082332630016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4949 % _timestamp‚1.7844961518183932e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89026128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291826664705536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4071043314323 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39236750 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26332617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39236752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26332613 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49789378 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379512433310368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4082243962880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89036287 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379512416271008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3308.420468075 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49799535 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383583459585331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4071130100861 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37538406 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49789378 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384043014620672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49799535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37538589 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39236752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287623625988608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287623531878912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4203038716928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11205793 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4082332630016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4949 % _timestamp‚1.7844961518183932e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89026128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291826664705536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4071043314323 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39236750 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26332617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39236752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26332613 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49789378 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379512433310368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4082243962880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89036287 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379512416271008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3308.420468075 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49799535 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383583459585331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4071130100861 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37538406 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49789378 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384043014620672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49799535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37538589 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39236752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287623625988608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287623531878912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4203038716928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291826443256320–,g]2¼- _wandbruntime‚3308  _runtime‚3308mCDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50272410 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4109585136957 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39610571 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387725365308928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391846156456960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11311590 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4120896847360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4242651162112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50272410 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11311776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391846156456960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26580361 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387262254191533 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391846262156288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391846262156288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4120808187392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚383152652015216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89893126 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290374280550912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4242780605952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387262150365747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37892137 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3341.307714843 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784496184705865e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89882981 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4120896847360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37891947 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚190 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚409 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26580357 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39610573 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387725365308928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚383152669054576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚186 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4120808187392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294616931713024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387725348269568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4109498350531 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39610573 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39610571 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50282553 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 trainer/global_step‚4999 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50272410 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4109585136957 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39610571 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387725365308928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391846156456960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11311590 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4120896847360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4242651162112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50272410 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11311776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391846156456960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26580361 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387262254191533 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391846262156288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391846262156288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4120808187392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚383152652015216 ø”²pM@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89893126 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290374280550912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4242780605952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387262150365747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37892137 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3341.307714843 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784496184705865e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89882981 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143™•8ä2É- _wandbruntime‚3341  _runtime‚3341$Äì&ÊæM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50282230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚3341.722080786 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4121682570240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387337328211637 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89900430 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50292387 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89910589 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50292387 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387800514975232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290430418278912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387800497935872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89900430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294674101795328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39618200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125242368 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚383226959579968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39618200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294673882443264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89910589 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4243464164352 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26585413 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4121682570240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37899112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391922197545472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4243589406720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391922197545472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39618202 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682:ëçQñ&ËëM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11313699 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚383226942540608 trainer/global_step‚4999 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50292387 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387800514975232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290430418278912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387800497935872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89900430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294674101795328 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39618200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125242368 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚383226959579968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39618200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294673882443264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89910589 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4243464164352 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26585413 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4121682570240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37899112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391922197545472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4243589406720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391922197545472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39618202 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50282230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 _runtime‚3341.722080786 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4121682570240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387337328211637 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89900430 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50292387 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89910589 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚395643095411712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784496217008995e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26833157 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚3373.610948594 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395642989712384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4160246787584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39992023 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297464574373376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [LDeviceSt*¹»*ÇatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50775405 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39992021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11419237 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4160158127616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚391015915166155 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4160246787584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90767428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚177 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293181166950912 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26833161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4148736881339 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4160158127616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11419064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297464355014144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395642989712384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚5049 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38252398 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38252221 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4283188063232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360áÌëÂæ'ØàO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50775405 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39992021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11419237 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4160158127616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚391015915166155 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4160246787584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90767428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚177 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293181166950912 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26833161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4148736881339 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4160158127616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚173 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11419064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297464355014144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395642989712384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚5049 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38252398 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38252221 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4283188063232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚395643095411712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784496217008995e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26833157 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚3373.610948594 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395642989712384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4160246787584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39992023 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297464574373376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Xˆ›2Ù- _wandbruntime‚3374  _runtime‚3374Üø§¡ì&ÚæM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395719030800896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50775082 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38259633 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297521296159744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386941485849568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293237304678912 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4160943843328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395718925094400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚391557998290432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39999650 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4149607162477 trainer/global_step‚5049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚414 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90784891 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4284120917504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚391090989186147 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚3374.024089029 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395719030800896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50775082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50785239 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚391557981251072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50785239 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293237398788608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26838217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38259457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297521519706112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚391557981251072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4160943843328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚391091093012045 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0H«n ÛëM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26838217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38259457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297521519706112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚391557981251072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4160943843328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚391091093012045 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395719030800896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50775082 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38259633 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297521296159744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386941485849568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293237304678912 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4160943843328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395718925094400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚391557998290432 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39999650 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4149607162477 trainer/global_step‚5049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚414 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90784891 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4284120917504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚391090989186147 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚3374.024089029 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395719030800896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50775082 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50785239 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚391557981251072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50785239 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293237398788608žĈ#D2Ü- _wandbruntime‚3374  _runtime‚3374øl?Ýÿ /loss/FRt1/assignment_loss‚0.952605030300853 2loss/SRqqt1/assignment_loss‚0.5995873540947321 /loss/FRt2/detection_loss‚0.5448832097651444 _step‚415 0loss/FRt2/assignment_loss‚2.0119123291899244 1loss/FBt2/assignment_loss‚0.03842472649888525 0loss/SRqqt2/detection_loss‚0.993878314793865 %loss/total_loss‚9.228482830339935 trainer/global_step‚5049 % _timestamp‚1.7844962174233828e+09 _runtime‚ 3374.02499373 2loss/SRqqt2/assignment_loss‚0.6593283578632336 0loss/SRqqt1/detection_loss‚0.656777416249527 /loss/FBt1/detection_loss‚0.5637002162342796 epoch‚12 3loss/FBt1/assignment_loss‚0.0011463581785769198 .loss/FRt1/detection_loss‚1.484596590210612 /loss/FBt2/detection_loss‚0.72164292696030123>9¿ Þ„ % _timestamp‚1.7844962174233828e+09 _runtime‚ 3374.02499373 2loss/SRqqt2/assignment_loss‚0.6593283578632336 0loss/SRqqt1/detection_loss‚0.656777416249527 /loss/FBt1/detection_loss‚0.5637002162342796 epoch‚12 3loss/FBt1/assignment_loss‚0.0011463581785769198 .loss/FRt1/detection_loss‚1.484596590210612 /loss/FBt2/detection_loss‚0.7216429269603012 /loss/FRt1/assignment_loss‚0.952605030300853 2loss/SRqqt1/assignment_loss‚0.5995873540947321 /loss/FRt2/detection_loss‚0.5448832097651444 _step‚415 0loss/FRt2/assignment_loss‚2.0119123291899244 1loss/FBt2/assignment_loss‚0.03842472649888525 0loss/SRqqt2/detection_loss‚0.993878314793865 %loss/total_loss‚9.228482830339935 trainer/global_step‚5049ŸC? 3ß‚(:¨ Ú€õҞ烿 disk.rbd0.out‚0& proc.memory.availableMB‚ 186.171875 cpu‚25.81471418819787 proc.cpu.threads‚17 network.recv‚916423 disk.md0.out‚ 89.37109375 disk.nvme0n1.in‚0" proc.memory.rssMB‚ 2961.7265625 network.sent‚ 7.376078e+06( disk./.usagePercent‚76.1721622359576" disk.dm-0.out‚28906.2119140625 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚21331.8857421875& disk.nvme1c1n1.out‚10307.244140625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚10307.244140625 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 2540.6875 disk.rbd0.in‚0% disk.nvme0n1.out‚21331.8857421875 disk.nvme1c1n1.in‚0$ memory_percent‚98.86369705200195) proc.memory.percent‚18.07694435119629$ disk./.usageGB‚1169.433120727539 disk.md0.in‚0}Ðבà‚(:† Ú€õÒ„Ê«¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.375' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]«NëAá‚(:¶ é€õÒÕ¦“¿" proc.memory.rssMB‚ 2961.7265625 cpu‚25.81354572966688 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚28907.23681640625! disk.dm-1.out‚2540.8232421875 disk.rbd0.out‚0) proc.memory.percent‚18.07694435119629) disk./.usagePercent‚76.17216496917173 disk.nvme0n1.in‚0& disk.nvme0n1.out‚21333.21533203125 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.20542240142822( proc.memory.availableMB‚ 130.18359375 network.sent‚ 7.440229e+06 disk.md0.in‚0( disk.nvme0c0n1.out‚21333.21533203125 proc.cpu.threads‚17 network.recv‚920766$ disk./.usageGB‚1169.433162689209' disk.nvme1c1n1.out‚10307.8798828125% disk.nvme1n1.out‚10307.8798828125 disk.md0.out‚ 89.74609375 disk.nvme0c0n1.in‚0*@sŸâ‚(:” é€õҶп¿ gpu.0.gpu‚88.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.349' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.449666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 :@Â9ã‚(:® ø€õÒŽ‡¿% disk./.usageGB‚1169.4006652832031 disk.dm-0.out‚28914.08203125) proc.memory.percent‚18.07694435119629 cpu‚25.812463341888318 proc.cpu.threads‚17( disk./.usagePercent‚76.1700482190655 disk.md0.in‚0 disk.md0.out‚ 90.23828125 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚21340.3642578125% disk.nvme1c1n1.out‚10308.73046875 network.recv‚923955 disk.nvme1n1.in‚0# disk.nvme1n1.out‚10308.73046875! disk.dm-1.out‚2540.9169921875 disk.nvme0n1.in‚0$ memory_percent‚99.20556545257568( proc.memory.availableMB‚ 130.16015625 network.sent‚ 7.445051e+06 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚21340.3642578125 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2961.7265625OõÙQœä‚(:‘ ø€õÒ˜‰¹¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.878' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.29266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ‡Ýz2å- _wandbruntime‚3406  _runtime‚3406^,W Èæ ' lr-AdamW/pg1‚0.00013875070784372646 ' lr-AdamW/pg2‚0.00013875070784372646 trainer/global_step‚5099 % _timestamp‚1.7844962496334674e+09 _runtime‚ 3406.23532196 _step‚416ÞýÁsÍçÇ _runtime‚ 3406.23532196 _step‚416 ' lr-AdamW/pg1‚0.00013875070784372646 ' lr-AdamW/pg2‚0.00013875070784372646 trainer/global_step‚5099 % _timestamp‚1.7844962496334674e+09 ws‘2è- _wandbruntime‚3406  _runtime‚3406ˆÔâ'éÜO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399439928667136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390581721593776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚395240331939328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40373471 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51268257 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91641730 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚417 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51258114 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4199508067840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40373473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295988147460608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399439822967808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4323879296512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91631585 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4188062198573 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4323760338432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40373473 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚395240331939328=®~xç'êáO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91631585 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4188062198573 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4323760338432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40373473 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚395240331939328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399439928667136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390581721593776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚395240331939328 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40373471 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51268257 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91641730 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚417 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51258114 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4199508067840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40373473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295988147460608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399439822967808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4323879296512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175¡N2ë- _wandbruntime‚3406  _runtime‚3406×coõìåM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91649034 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4200293783552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38619487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390656012119168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394844753986555 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399515758349824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51278091 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4324688105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 _step‚418 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38619309 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300368973294080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4324564960256 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390655995079808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4200382450688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91659193 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40381100 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27091017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27091013 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394844857812453 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51278091 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4200293783552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4200382450688 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11528296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91659193 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.784496250050743e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91649034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚3406.652743592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40381102 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40381100 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51267934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51267934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300368756039168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296044285188608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚395315481605632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚395315481605632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40381102 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4188845693285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399515864056320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11528470 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399515864056320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812ýÁ¸ð&íêM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4200382450688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91659193 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40381100 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27091017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27091013 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394844857812453 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51278091 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4200293783552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4200382450688 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11528296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91659193 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.784496250050743e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91649034 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 _runtime‚3406.652743592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40381102 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40381100 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51267934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51267934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300368756039168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296044285188608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚395315481605632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚395315481605632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40381102 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4188845693285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399515864056320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11528470 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399515864056320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91649034 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4200293783552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38619487 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390656012119168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394844753986555 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399515758349824 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51278091 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4324688105472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 _step‚418 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38619309 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300368973294080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4324564960256 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390655995079808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760¢Ÿ£‘2î- _wandbruntime‚3406  _runtime‚3406¸PôÅï‚ 1loss/SRqqt2/detection_loss‚0.9880033213498484 /loss/FBt1/detection_loss‚0.5918487883905609 %loss/total_loss‚9.030091261082154 _runtime‚3406.653295833 1loss/FBt2/assignment_loss‚0.10067812370415784 0loss/FBt2/detection_loss‚0.49814794674791674 $ _timestamp‚1.784496250051785e+09 epoch‚12 2loss/SRqqt2/assignment_loss‚0.7022286731768677 _step‚419 0loss/FRt1/assignment_loss‚0.7744423981755972 /loss/FRt2/detection_loss‚0.5776010466410528 1loss/SRqqt1/detection_loss‚0.6658406131031277 trainer/global_step‚5099 /loss/FRt2/assignment_loss‚2.035373521199919 2loss/SRqqt1/assignment_loss‚0.6295794305114828 2loss/FBt1/assignment_loss‚0.013100289302589942 /loss/FRt1/detection_loss‚1.4532471087790326ò–»f ð‡ 2loss/FBt1/assignment_loss‚0.013100289302589942 /loss/FRt1/detection_loss‚1.4532471087790326 1loss/SRqqt2/detection_loss‚0.9880033213498484 /loss/FBt1/detection_loss‚0.5918487883905609 %loss/total_loss‚9.030091261082154 _runtime‚3406.653295833 1loss/FBt2/assignment_loss‚0.10067812370415784 0loss/FBt2/detection_loss‚0.49814794674791674 $ _timestamp‚1.784496250051785e+09 epoch‚12 2loss/SRqqt2/assignment_loss‚0.7022286731768677 _step‚419 0loss/FRt1/assignment_loss‚0.7744423981755972 /loss/FRt2/detection_loss‚0.5776010466410528 1loss/SRqqt1/detection_loss‚0.6658406131031277 trainer/global_step‚5099 /loss/FRt2/assignment_loss‚2.035373521199919 2loss/SRqqt1/assignment_loss‚0.6295794305114828£‚@ÿˆ(ñ‚(: ‡õÒÙè·¿ disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚90.4375) proc.memory.percent‚18.07694435119629# disk.nvme1c1n1.out‚ 10309.296875 disk.dm-1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1169.4013595581055 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 28914.6875 cpu‚25.81138439000093 network.sent‚ 7.460816e+06' disk.nvme0c0n1.out‚21341.0380859375 disk.nvme0n1.in‚0% disk.nvme0n1.out‚21341.0380859375! disk.dm-1.out‚2541.1025390625$ memory_percent‚99.20730590820312# proc.memory.availableMB‚129.875" proc.memory.rssMB‚ 2961.7265625! disk.nvme1n1.out‚ 10309.296875 network.recv‚927494) disk./.usagePercent‚76.17009344133564Úä³ò‚(:’ ‡õÒäŠç¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.675' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.224999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0An§—4ó‚(:© –õÒ©¤ƒ¿ disk.nvme0c0n1.in‚0( proc.memory.availableMB‚ 129.87109375) proc.memory.percent‚18.07694435119629$ disk.nvme1n1.out‚10309.912109375" proc.memory.rssMB‚ 2961.7265625 cpu‚25.810254022998933 proc.cpu.threads‚17) disk./.usagePercent‚76.17007878136894% disk./.usageGB‚1169.4011344909668! disk.nvme0n1.out‚ 21343.015625& disk.nvme1c1n1.out‚10309.912109375 disk.md0.in‚0! disk.dm-1.out‚2541.1572265625 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 7.510362e+06 network.recv‚931649 disk.nvme0n1.in‚0 disk.md0.out‚ 90.734375 disk.dm-0.in‚0" disk.dm-0.out‚28916.5888671875# disk.nvme0c0n1.out‚ 21343.015625$ memory_percent‚99.20732975006104 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-1.in‚0\£¬œô‚(:‘ –õÒ£ÙÀ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚221.226' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.74199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈìL2õ- _wandbruntime‚3438  _runtime‚3438“DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403236656223232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4238858008064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4364428905984 % _timestamp‚1.7844962819666133e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40754923 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4238858008064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27338757 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4238946668032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27338761 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4227213942955 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403236761922560 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4364301559296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 trainer/global_step‚5149 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403236656223232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40754921 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398523444766971 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40754921 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51750966 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38972371 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298795033860608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394296230824016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 _runtime‚3438.568475438 _step‚421 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92505887 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51761109 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4238946668032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403236761922560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398997815254528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92516032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40754923 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298794939750912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398997815254528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398997798215168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303159462766592 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303159241310208 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38972558 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11633797AJÙú úáO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27338761 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4227213942955 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403236761922560 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4364301559296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 trainer/global_step‚5149 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403236656223232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40754921 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398523444766971 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40754921 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51750966 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38972371 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298795033860608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394296230824016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 _runtime‚3438.568475438 _step‚421 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92505887 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51761109 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4238946668032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403236761922560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚183 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398997815254528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92516032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40754923 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298794939750912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398997815254528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398997798215168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303159462766592 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303159241310208 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38972558 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11633797 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4227300729381 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51761109 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚394296247863376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403236656223232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4238858008064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4364428905984 % _timestamp‚1.7844962819666133e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40754923 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4238858008064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27338757 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4238946668032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¥>oÄx2û- _wandbruntime‚3438  _runtime‚3438Ó;9Øì&üæM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4239732390912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92523336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7844962823802469e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51770943 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403312697311744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4239643723776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚399072947881472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303216181497344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27343817 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27343813 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298851077478912 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51760786 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38979738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40762552 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4228084224093 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4227997437555 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38979547 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303216411335168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229837824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40762550 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40762552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3438.982219035 _step‚422 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40762550 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚191 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4365239746560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92523336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚399072947881472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92533495 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11635734 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4239643723776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398598622612861 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚399072964920832ÕÜðòñ&ýëM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚394370538388768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38979738 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40762552 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4228084224093 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4227997437555 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38979547 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚303216411335168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229837824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40762550 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40762552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3438.982219035 _step‚422 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40762550 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚191 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4365239746560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92523336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚399072947881472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92533495 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11635734 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4239643723776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398598622612861 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚399072964920832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4239732390912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92523336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 % _timestamp‚1.7844962823802469e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51770943 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403312697311744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4239643723776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚399072947881472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303216181497344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27343817 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27343813 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298851077478912 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51760786 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _runtime‚3471.486515311 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4278296608256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41136371 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39331676 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11740115 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398010774132976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407033489478656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398010757093616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52253961 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306006849592320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402755298569728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301601920260608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4404795693568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39331490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52243818 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402277209567379 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306006621844480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41136373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4278207948288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4278296608256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4278207948288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 trainer/global_step‚5199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52253961 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4266539260189 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41136373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27591561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93380189 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407033595177984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407033489478656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93390334 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227747840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4404929331712 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402755298569728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4266452473763 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402755298569728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4266452473763 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _runtime‚3471.486515311 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4278296608256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41136371 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39331676 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11740115 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398010774132976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407033489478656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398010757093616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52253961 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306006849592320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402755298569728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301601920260608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4404795693568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39331490 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52243818 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402277209567379 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306006621844480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41136373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4278207948288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4278296608256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4278207948288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 trainer/global_step‚5199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52253961 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4266539260189 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41136373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27591561 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93380189 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407033595177984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407033489478656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93390334 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227747840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821©D×>´2‹- _wandbruntime‚3471  _runtime‚3471›.P‚ì&ŒæM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52263795 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306063784832512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4267235968363 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39338823 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402352283587371 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93407797 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398085047619008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11742038 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39338651 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301658057988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93397638 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93397638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352064 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27596613 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407109424860672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125242368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402830431196672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0{; g¸ ëM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93407797 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398085047619008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11742038 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39338651 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301658057988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_˜g5Å9pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93397638 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93397638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219352064 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27596613 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407109424860672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 125242368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402830431196672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52263795 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306063784832512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4267235968363 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39338823 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402352283587371 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812ªØN•k2Ž- _wandbruntime‚3471  _runtime‚3471L^”D 0loss/FRt1/assignment_loss‚0.7946235750676223 1loss/SRqqt2/detection_loss‚0.9703057726098911 %loss/total_loss‚8.833446800403763 0loss/FRt2/assignment_loss‚1.9981246619992292 1loss/FBt2/assignment_loss‚0.03029956123716309 1loss/SRqqt1/detection_loss‚0.6130286546955566 /loss/FBt2/detection_loss‚0.7368375953907768 epoch‚12 2loss/SRqqt2/assignment_loss‚0.5886292934868697 2loss/FBt1/assignment_loss‚0.007030681841302681 trainer/global_step‚5199 % _timestamp‚1.7844963153030286e+09 _step‚427 2loss/SRqqt1/assignment_loss‚0.5762286850375072 .loss/FRt1/detection_loss‚1.485393115681271 -loss/FRt2/detection_loss‚0.60225489591168 0loss/FBt1/detection_loss‚0.43069030744489284 _runtime‚3471.904590818$ƒÓ † _runtime‚3471.904590818 0loss/FRt1/assignment_loss‚0.7946235750676223 1loss/SRqqt2/detection_loss‚0.9703057726098911 %loss/total_loss‚8.833446800403763 0loss/FRt2/assignment_loss‚1.9981246619992292 1loss/FBt2/assignment_loss‚0.03029956123716309 1loss/SRqqt1/detection_loss‚0.6130286546955566 /loss/FBt2/detection_loss‚0.7368375953907768 epoch‚12 2loss/SRqqt2/assignment_loss‚0.5886292934868697 2loss/FBt1/assignment_loss‚0.007030681841302681 trainer/global_step‚5199 % _timestamp‚1.7844963153030286e+09 _step‚427 2loss/SRqqt1/assignment_loss‚0.5762286850375072 .loss/FRt1/detection_loss‚1.485393115681271 -loss/FRt2/detection_loss‚0.60225489591168 0loss/FBt1/detection_loss‚0.43069030744489284«z“5=‘‚(:² ÃõÒØ·“¿ disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚21347.447265625 disk.dm-0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚99.20644760131836# proc.memory.rssMB‚ 2961.73046875 cpu‚25.814126986914488 network.sent‚ 7.594434e+06) disk./.usagePercent‚76.17016922590923% disk./.usageGB‚1169.4025230407715& disk.nvme1n1.out‚10313.34033203125! disk.dm-0.out‚28920.201171875 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚942238 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚21347.447265625( disk.nvme1c1n1.out‚10313.34033203125& proc.memory.availableMB‚ 130.015625( proc.memory.percent‚18.0769681930542 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 92.1171875 disk.dm-1.in‚0" disk.dm-1.out‚2542.44580078125 disk.rbd0.out‚0ùý(¿’‚(:„ ÃõÒèÙÆ¿ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.09' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Úk2“- _wandbruntime‚3485  _runtime‚3485yh©=”£{ 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8479690551757812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943860173225403 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5051490664482117 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45901599526405334 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09362135082483292 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9935522675514221 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6431736946105957 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11906550079584122 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985470175743103 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6891393065452576 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863508343696594 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6517893671989441 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948678016662598 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5646941661834717 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10843639075756073 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12271308898925781 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4698401987552643 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32968956232070923 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08883547782897949 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6629273295402527 *jet/accuracy_1_of_4‚0.9397470355033875 jet/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13744406402111053 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951111674308777 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_1_of_1‚0.27584171295166016 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24567247927188873 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8077853918075562 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32691553235054016 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5790236592292786 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5863659977912903 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961382746696472 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6013340353965759 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5408333539962769 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011414452455937862 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 _step‚428 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900347590446472 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10628999024629593 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07054240256547928 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8037793040275574 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17083333432674408 /particle/accuracy_4_of_4‚0.9662081599235535 (particle/accuracy‚0.8430951237678528 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21453523635864258 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7224029302597046 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6629273295402527 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13691702485084534 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6540354490280151 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7257747650146484 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5339730978012085 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45859473943710327 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6745472550392151 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5935152173042297 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5899666547775269 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5573655962944031 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8105064630508423 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022718895226716995 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015542024746537209 particle/accuracy_1_of_1‚1 $ _timestamp‚1.784496329009685e+09 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13846319913864136 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.987639844417572 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5674188733100891 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07437397539615631 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 trainer/global_step‚5199 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.796976625919342 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5509737730026245 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )loss/val_total_loss‚8.939272869757204 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.060711730271577835 epoch‚12 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9199349284172058 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005990279372781515 /particle/accuracy_3_of_4‚0.9965986609458923 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13714472949504852 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.359375 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029786504805088043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5169059038162231 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26308754086494446 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6458333730697632 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21503305435180664 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5625953674316406 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5194975733757019 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.539143979549408 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1180892139673233 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.047001879662275314 +jet/accuracy_2_of_2‚0.11405742913484573 &particle/f_score‚0.631616473197937 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908613562583923 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9886273145675659 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_2_of_4‚0.5477045774459839 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8924985527992249 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995329737663269 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14311763644218445 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7238565683364868 +particle/sensitivity‚0.6263968348503113 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14413413405418396 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6124835014343262 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.8827838897705078 jet/accuracy_1_of_5‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5214506387710571 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17460109293460846 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5444201231002808 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0658961683511734 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09768830239772797 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5666460394859314 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975658655166626 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_3‚0.3963993191719055 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9963806867599487 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8716952204704285 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5627188682556152 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2552984654903412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3402777910232544 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6177643537521362 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6040418744087219 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7407407164573669 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10619413107633591 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6152709126472473 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06517075002193451 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12213905900716782 +particle/specificity‚0.9023550152778625 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15307503938674927 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919494390487671 7validation_average_jet_accuracy‚0.31753891706466675 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.932620108127594 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3984339237213135 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1586836874485016 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5674188733100891 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1569673717021942 _runtime‚3485.611941929 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6719818115234375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6894009709358215 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5556283593177795 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.807634174823761 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.047001879662275314 *jet/accuracy_1_of_2‚0.7196664810180664 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5584250688552856 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9772543907165527 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01205478236079216 +jet/accuracy_3_of_3‚0.06833399832248688 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1223435252904892 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7261878252029419 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5943335890769958 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6104044914245605 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09062311053276062 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4166666865348816 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.060301847755908966 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6215046644210815 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1565074473619461 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14311763644218445 particle/accuracy_2_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23820599913597107 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05“(Ì4œ•¨{ & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961382746696472 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6013340353965759 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5408333539962769 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011414452455937862 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 _step‚428 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900347590446472 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10628999024629593 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07054240256547928 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8037793040275574 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17083333432674408 /particle/accuracy_4_of_4‚0.9662081599235535 (particle/accuracy‚0.8430951237678528 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21453523635864258 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7224029302597046 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6629273295402527 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13691702485084534 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6540354490280151 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7257747650146484 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5339730978012085 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45859473943710327 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6745472550392151 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5935152173042297 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5899666547775269 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5573655962944031 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8105064630508423 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022718895226716995 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015542024746537209 particle/accuracy_1_of_1‚1 $ _timestamp‚1.784496329009685e+09 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13846319913864136 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.987639844417572 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5674188733100891 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07437397539615631 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 trainer/global_step‚5199 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.796976625919342 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5509737730026245 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )loss/val_total_loss‚8.939272869757204 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.060711730271577835 epoch‚12 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9199349284172058 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005990279372781515 /particle/accuracy_3_of_4‚0.9965986609458923 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13714472949504852 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.359375 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029786504805088043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5169059038162231 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26308754086494446 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6458333730697632 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21503305435180664 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5625953674316406 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5194975733757019 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.539143979549408 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1180892139673233 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.047001879662275314 +jet/accuracy_2_of_2‚0.11405742913484573 &particle/f_score‚0.631616473197937 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908613562583923 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9886273145675659 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_2_of_4‚0.5477045774459839 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8924985527992249 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995329737663269 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14311763644218445 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7238565683364868 +particle/sensitivity‚0.6263968348503113 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14413413405418396 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6124835014343262 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.8827838897705078 jet/accuracy_1_of_5‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5214506387710571 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17460109293460846 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5444201231002808 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0658961683511734 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09768830239772797 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5666460394859314 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975658655166626 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_oY!0Ç#f_3‚0.3963993191719055 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9963806867599487 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8716952204704285 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5627188682556152 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2552984654903412 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3402777910232544 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6177643537521362 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6040418744087219 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7407407164573669 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10619413107633591 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6152709126472473 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06517075002193451 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12213905900716782 +particle/specificity‚0.9023550152778625 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15307503938674927 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919494390487671 7validation_average_jet_accuracy‚0.31753891706466675 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.932620108127594 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3984339237213135 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1586836874485016 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5674188733100891 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1569673717021942 _runtime‚3485.611941929 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6719818115234375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6894009709358215 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5556283593177795 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.807634174823761 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.047001879662275314 *jet/accuracy_1_of_2‚0.7196664810180664 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5584250688552856 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9772543907165527 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01205478236079216 +jet/accuracy_3_of_3‚0.06833399832248688 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1223435252904892 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7261878252029419 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5943335890769958 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6104044914245605 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09062311053276062 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4166666865348816 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.060301847755908966 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6215046644210815 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1565074473619461 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14311763644218445 particle/accuracy_2_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23820599913597107 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8479690551757812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943860173225403 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5051490664482117 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45901599526405334 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09362135082483292 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9935522675514221 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6431736946105957 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11906550079584122 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985470175743103 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6891393065452576 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863508343696594 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6517893671989441 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948678016662598 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5646941661834717 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10843639075756073 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12271308898925781 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4698401987552643 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32968956232070923 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08883547782897949 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6629273295402527 *jet/accuracy_1_of_4‚0.9397470355033875 jet/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13744406402111053 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951111674308777 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_1_of_1‚0.27584171295166016 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24567247927188873 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8077853918075562 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32691553235054016 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5790236592292786 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5863659977912903 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1¬ý±#˜B–‚(:· ÒõÒÕî·¿ proc.cpu.threads‚17" disk.dm-1.out‚2542.56298828125 network.sent‚ 7.657042e+06 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10314.31689453125 disk.md0.out‚ 92.55078125 disk.dm-0.in‚0( disk.nvme0c0n1.out‚21349.69677734375 disk.nvme0n1.in‚0) disk./.usagePercent‚76.17016077779283 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚99.20692443847656# proc.memory.rssMB‚ 2961.73046875# disk.dm-0.out‚28922.38818359375 network.recv‚946837% disk./.usageGB‚1169.4023933410645& disk.nvme1n1.out‚10314.31689453125 cpu‚25.818622053714268 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚21349.69677734375$ proc.memory.availableMB‚129.9375( proc.memory.percent‚18.0769681930542}2 ž—‚(:“ ÒõÒô÷ð¿ gpu.0.gpu‚95.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚231.013' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.00433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã/Ëp@˜‚(:µ áõÒ½÷ç¾ disk.nvme0n1.in‚0 disk.dm-1.out‚2542.677734375# proc.memory.rssMB‚ 2961.73046875 network.sent‚ 7.702871e+06 network.recv‚951192% disk./.usageGB‚1169.4023361206055( disk.nvme0c0n1.out‚21355.31298828125# memory_percent‚99.2069959640503( proc.memory.availableMB‚ 129.92578125 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚21355.31298828125 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚10314.7744140625% disk.nvme1n1.out‚10314.7744140625 disk.dm-0.in‚0 disk.dm-1.in‚0( proc.memory.percent‚18.0769681930542 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 92.77734375# disk.dm-0.out‚28927.84130859375 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.81763008366884 proc.cpu.threads‚17) disk./.usagePercent‚76.17015705068265hrüÚ“™‚(:ˆ áõÒêÔ¿ gpu.0.gpu‚63.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚217.515' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚72.505 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÝ4”2š- _wandbruntime‚3518  _runtime‚3518ôÎN9ɛà ' lr-AdamW/pg1‚0.00013741466800301294 ' lr-AdamW/pg2‚0.00013741466800301294 trainer/global_step‚5249 % _timestamp‚1.7844963620716393e+09 _runtime‚3518.674298282 _step‚429^ÎeáÎœÈ trainer/global_step‚5249 % _timestamp‚1.7844963620716393e+09 _runtime‚3518.674298282 _step‚429 ' lr-AdamW/pg1‚0.00013741466800301294 ' lr-AdamW/pg2‚0.00013741466800301294­6^Ž2- _wandbruntime‚3518  _runtime‚3518æ=â'žÜO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7844963620725138e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4332987262976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11894099 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94458793 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52829315 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11894270 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402698767438768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚402698750399408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚407509269180928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4333075922944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52819172 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚407509252141568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407509269180928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41639621 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305212171865600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411842239404544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309673246484480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27952612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41639623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39846886 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411842345103872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4332987262976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚171 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11894099 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94458793 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52829315 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11894270 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402698767438768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚402698750399408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚407509269180928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4333075922944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52819172 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚407509252141568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚407509269180928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41639621 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 _step‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305212171865600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411842239404544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309673246484480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27952612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41639623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39846886 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411842345103872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚407095034508430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305268309593600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚431 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41647250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4333772978688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39853909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 % _timestamp‚1.7844963624894347e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41647250 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407584418847232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52839149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚407094930682532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407584401807872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94476242 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4321889757732 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41647252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94476242 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4321976544270 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94486401 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309729983696384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305268215483904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407584418847232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27957672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3519.091509867 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39854095 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4333772978688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚411918280493056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚407095034508430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305268309593600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚431 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41647250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4333772978688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39853909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 % _timestamp‚1.7844963624894347e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41647250 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407584418847232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52839149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚407094930682532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407584401807872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94476242 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4321889757732 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41647252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94476242 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4321976544270 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94486401 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309729983696384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305268215483904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407584418847232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27957672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3519.091509867 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39854095 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4333772978688¯ÖöZÚ2£- _wandbruntime‚3519  _runtime‚3519ú±»¤€ 1loss/SRqqt1/detection_loss‚0.6677217842638359 1loss/SRqqt2/detection_loss‚1.0607335515961491 /loss/FRt2/assignment_loss‚2.060031889763075 2loss/SRqqt2/assignment_loss‚0.6969678700035525 .loss/FBt2/detection_loss‚0.631103722210757 trainer/global_step‚5249 % _timestamp‚1.7844963624905674e+09 _step‚432 2loss/SRqqt1/assignment_loss‚0.6282048582455398 1loss/FBt2/assignment_loss‚0.26158100824343006 /loss/FRt1/detection_loss‚1.4435616859676625 /loss/FRt2/detection_loss‚0.5299134442224704 /loss/FBt1/detection_loss‚0.5451849290870101 %loss/total_loss‚9.360914513737118 _runtime‚3519.092292478 epoch‚13 0loss/FRt1/assignment_loss‚0.8193160601465119 1loss/FBt1/assignment_loss‚0.01659370998712454ö²äº ¥… /loss/FBt1/detection_loss‚0.5451849290870101 %loss/total_loss‚9.360914513737118 _runtime‚3519.092292478 epoch‚13 0loss/FRt1/assignment_loss‚0.8193160601465119 1loss/FBt1/assignment_loss‚0.01659370998712454 1loss/SRqqt1/detection_loss‚0.6677217842638359 1loss/SRqqt2/detection_loss‚1.0607335515961491 /loss/FRt2/assignment_loss‚2.060031889763075 2loss/SRqqt2/assignment_loss‚0.6969678700035525 .loss/FBt2/detection_loss‚0.631103722210757 trainer/global_step‚5249 % _timestamp‚1.7844963624905674e+09 _step‚432 2loss/SRqqt1/assignment_loss‚0.6282048582455398 1loss/FBt2/assignment_loss‚0.26158100824343006 /loss/FRt1/detection_loss‚1.4435616859676625 /loss/FRt2/detection_loss‚0.5299134442224704°D€?;¦‚(:° ðõÒùÖº¿( disk.nvme1c1n1.out‚10315.56298828125 disk.nvme1n1.in‚0# disk.dm-0.out‚28930.31396484375& disk.nvme1n1.out‚10315.56298828125 proc.cpu.threads‚17) disk./.usagePercent‚76.17017195912335 disk.nvme0n1.in‚0# memory_percent‚99.2072582244873' proc.memory.availableMB‚ 129.8828125# proc.memory.rssMB‚ 2961.73046875( proc.memory.percent‚18.0769681930542 cpu‚25.81662712333574 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚21357.84765625 network.recv‚954631% disk./.usageGB‚1169.4025650024414 disk.md0.out‚ 93.13671875 disk.dm-1.in‚0 disk.dm-1.out‚2542.763671875 disk.rbd0.in‚0 network.sent‚ 7.718636e+06 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚21357.84765625 disk.nvme1c1n1.in‚0“4÷R“§‚(:ˆ ðõÒ’…ï¿ gpu.0.gpu‚76.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚219.981' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.327 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ'‘A<¨‚(:± ÿõÒ±”ì¾ disk.nvme1n1.in‚0# disk.nvme1n1.out‚10316.31640625 disk.dm-0.in‚0 proc.cpu.threads‚17 network.recv‚958946( disk.nvme0c0n1.out‚21358.56884765625% disk.nvme1c1n1.out‚10316.31640625 disk.md0.out‚ 93.3828125 disk.dm-1.in‚0' proc.memory.availableMB‚ 129.8828125 network.sent‚ 7.769874e+06% disk./.usageGB‚1169.4028434753418 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0! disk.dm-1.out‚2542.9052734375 disk.rbd0.in‚0# memory_percent‚99.2072582244873( proc.memory.percent‚18.0769681930542& disk.nvme0n1.out‚21358.56884765625# disk.dm-0.out‚28931.11083984375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2961.73046875 cpu‚25.815581430365807 disk.md0.in‚0) disk./.usagePercent‚76.17019009772622 disk.nvme1c1n1.in‚0%z£†›©‚(: ÿõÒ÷¡­À gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚221.99' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.99666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä+p2ª- _wandbruntime‚3551  _runtime‚3551yýš Ç«Á trainer/global_step‚5299 % _timestamp‚1.7844963949075727e+09 _runtime‚3551.510268576 _step‚433 & lr-AdamW/pg1‚0.0001369619366567541 & lr-AdamW/pg2‚0.0001369619366567541°$ZÄÌ¬Æ % _timestamp‚1.7844963949075727e+09 _runtime‚3551.510268576 _step‚433 & lr-AdamW/pg1‚0.0001369619366567541 & lr-AdamW/pg2‚0.0001369619366567541 trainer/global_step‚5299±ê 2­- _wandbruntime‚3551  _runtime‚3551ðúÕ‰Ì ®ÜO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312520730727936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411266752496128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406413293708368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4372337203200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53322167 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚5299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42021073 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415639178359296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4372337203200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚410773725288734 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4360344793940 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40208103 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚410773621462948 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40207915 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4372425863168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7844963949082742e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415639072659968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411266752496128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42021071 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415639072659968 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411266735456768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406413276669008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312520505077248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocaêk©tion.all.allocated‚95343240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308018964155904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12002687 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411266735456768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53312024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4501540921344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0#y‹•ç'¯áO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12002687 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411266735456768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53312024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4501540921344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312520730727936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411266752496128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406413293708368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4372337203200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53322167 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 trainer/global_step‚5299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42021073 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415639178359296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4372337203200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚410773725288734 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4360344793940 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40208103 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚410773621462948 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40207915 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4372425863168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 % _timestamp‚1.7844963949082742e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415639072659968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411266752496128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42021071 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415639072659968 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411266735456768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406413276669008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312520505077248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95343240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308018964155904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426²àÂéí2°- _wandbruntime‚3551  _runtime‚3551â$\wì&±æM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42028702 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95350544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7844963953202448e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410848799308838 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4373122918912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415715113748480 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410848695482940 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53321844 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _runtime‚3551.922289519 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40215266 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95360703 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53332001 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4361128288540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42028700 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312577454504960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚435 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95360703 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42028702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42028700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308075195993600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚308075101883904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411341902162432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4373122918912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411341885123072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312577671759872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40215084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28210472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28210468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12004794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12004616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4361215075078 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411341885123072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4373211586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53332001 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488º~gñ&²ëM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411341902162432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4373122918912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411341885123072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312577671759872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40215084 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28210472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28210468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12004794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12004616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4361215075078 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411341885123072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4373211586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53332001 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42028702 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95350544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7844963953202448e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410848799308838 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4373122918912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415715113748480 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217254912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410848695482940 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53321844 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _runtime‚3551.922289519 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40215266 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95360703 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53332001 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4361128288540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42028700 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312577454504960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚435 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95360703 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42028702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42028700 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308075195993600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚308075101883904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0³”µp§2³- _wandbruntime‚3551  _runtime‚3551…ïó}‘´‚ % _timestamp‚1.7844963953212876e+09 _runtime‚ 3551.92283341 0loss/FRt1/assignment_loss‚0.8734134714214169 2loss/FBt1/assignment_loss‚0.011152057775204213 1loss/SRqqt2/detection_loss‚1.0650909494256342 /loss/FBt2/detection_loss‚0.7596509823237326 2loss/SRqqt1/assignment_loss‚0.6358997610246973 1loss/SRqqt2/assignment_loss‚0.766170943864732 /loss/FRt1/detection_loss‚1.5024040115274946 0loss/FBt1/detection_loss‚0.41498282265168845 trainer/global_step‚5299 _step‚436 0loss/FRt2/assignment_loss‚1.9484319979461229 1loss/FBt2/assignment_loss‚0.04670226565707765 /loss/FRt2/detection_loss‚0.5G#w946107712302031 1loss/SRqqt1/detection_loss‚0.6567550277527215 %loss/total_loss‚9.275265062600726 epoch‚13“1Ø µ‡ _runtime‚ 3551.92283341 0loss/FRt1/assignment_loss‚0.8734134714214169 2loss/FBt1/assignment_loss‚0.011152057775204213 1loss/SRqqt2/detection_loss‚1.0650909494256342 /loss/FBt2/detection_loss‚0.7596509823237326 2loss/SRqqt1/assignment_loss‚0.6358997610246973 1loss/SRqqt2/assignment_loss‚0.766170943864732 /loss/FRt1/detection_loss‚1.5024040115274946 0loss/FBt1/detection_loss‚0.41498282265168845 trainer/global_step‚5299 _step‚436 0loss/FRt2/assignment_loss‚1.9484319979461229 1loss/FBt2/assignment_loss‚0.04670226565707765 /loss/FRt2/detection_loss‚0.5946107712302031 1loss/SRqqt1/detection_loss‚0.6567550277527215 %loss/total_loss‚9.275265062600726 epoch‚13 % _timestamp‚1.7844963953212876e+09´uyˆ‰ž¶‚(:“ Ž‚õҚì¿ gpu.0.gpu‚66.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.30733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÌ8•?·‚(:´ Ž‚õÒ²³¡À network.sent‚ 7.785639e+06 network.recv‚962060& disk.nvme0n1.out‚21381.72900390625' disk.nvme1c1n1.out‚10317.0419921875# disk.dm-0.out‚28954.13916015625$ memory_percent‚99.20763969421387# proc.memory.rssMB‚ 2961.73046875 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.dm-1.out‚2543.0458984375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21381.72900390625 disk.nvme1n1.in‚0 disk.md0.out‚ 93.703125 disk.dm-0.in‚0' proc.memory.availableMB‚ 129.8203125) disk./.usagePercent‚76.11715282296691 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0( proc.memory.percent‚18.0769681930542 cpu‚25.81472392017593% disk./.usageGB‚1168.5885887145996% disk.nvme1n1.out‚10317.0419921875šÙH!C¸‚(:¸ ‚õÒ²œ‰¿( disk.nvme1c1n1.out‚10317.77294921875 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10317.77294921875 disk.dm-0.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚21382.68017578125( proc.memory.availableMB‚ 129.81640625 network.recv‚966589% disk./.usageGB‚1168.5884552001953$ memory_percent‚99.20766353607178( proc.memory.percent‚18.0769681930542 cpu‚25.813843176604312 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 2961.73046875 proc.cpu.threads‚17 network.sent‚ 7.836954e+06 disk.md0.out‚ 94.109375# disk.dm-0.out‚28954.76806640625 disk.dm-1.in‚0" disk.dm-1.out‚2543.23583984375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21382.68017578125 disk.nvme0n1.in‚0( disk./.usagePercent‚76.1171441263765÷¬œž¹‚(:“ ‚õÒ×˸¿ gpu.0.gpu‚91.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚224.717' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.90566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'p°2º- _wandbruntime‚3583  _runtime‚3583À ”ÍÇ»Á _runtime‚3583.794906929 _step‚437 & lr-AdamW/pg1‚0.0001365055612107222 & lr-AdamW/pg2‚0.0001365055612107222 trainer/global_step‚5349 % _timestamp‚1.7844964271930609e+09CxœÌ¼Æ _step‚437 & lr-AdamW/pg1‚0.0001365055612107222 & lr-AdamW/pg2‚0.0001365055612107222 trainer/global_step‚5349 % _timestamp‚1.7844964271930609e+09 _runtime‚3583.794906929µÜѪ2½- _wandbruntime‚3583  _runtime‚35838ÓÕCâ'¾ÜO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42402523 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419435905915392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415024235811328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53804876 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5349 % _timestamp‚1.7844964271940744e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419436011614720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419435905915392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4411775803392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415024235811328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415024218771968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚414527490089142 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53804876 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚191 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96207397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12110556 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4542111860736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40568768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42402523 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53815019 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53815019 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315368183873024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315367962416640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410127802938608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42402521 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42402521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _runtime‚3583.795876091üE¨»ç'¿áO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40568768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42402523 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53815019 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53815019 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315368183873024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315367962416640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410127802938608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42402521 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42402521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _runtime‚3583.795876091 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42402523 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419435905915392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415024235811328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53804876 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5349 % _timestamp‚1.7844964271940744e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419436011614720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419435905915392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4411775803392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415024235811328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415024218771968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚414527490089142 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53804876 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚191 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96207397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12110556 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4542111860736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688¶Ÿ«Ö2À- _wandbruntime‚3584  _runtime‚35848f‚Nj ÁæM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53824853 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4543048540672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42410150 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40576105 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96235005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4412561526272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚167 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _step‚439 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4412472859136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4412561526272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415099368438272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96235005 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96224846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12112666 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410202110503360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚414602460283348 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4400453605886 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42410152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415099385477632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419511947003904 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419511841297408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96224846 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315424909485056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310882082393600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4400366819348 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28463272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4542921201152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3584.208092253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310881988283904 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ZFDeviceStatsMonitor.on_train_batch_end/requesteçA9‚d_bytes.large_pool.freed‚410202093464000 % _timestamp‚1.7844964276060882e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40575934 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414602564109246 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812ͼïXñ&ÂëM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53824853 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4543048540672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42410150 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40576105 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96235005 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4412561526272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚167 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚171 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 _step‚439 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4412472859136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4412561526272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415099368438272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96235005 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96224846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12112666 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410202110503360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚414602460283348 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4400453605886 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42410152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415099385477632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419511947003904 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419511841297408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96224846 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315424909485056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310882082393600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4400366819348 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28463272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4542921201152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3584.208092253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310881988283904 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410202093464000 % _timestamp‚1.7844964276060882e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40575934 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414602564109246 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0·ëx2œ2Ã- _wandbruntime‚3584  _runtime‚3584¬†† Ä„ trainer/global_step‚5349 % _timestamp‚1.7844964276071002e+09 epoch‚13 0loss/FRt1/assignment_loss‚0.9407352674980552 0loss/FRt2/assignment_loss‚2.0305763549538725 2loss/SRqqt1/assignment_loss‚0.6282078855631497 2loss/SRqqt2/assignment_loss‚0.7194649030670542 /loss/FRt1/detection_loss‚1.5626760176487495 /loss/FRt2/detection_loss‚0.5740782634176026 1loss/SRqqt1/detection_loss‚0.7149391920071968 1loss/FBt2/assignment_loss‚0.17510880404180815 %loss/total_loss‚9.535838749176737 1loss/SRqqt2/detection_loss‚1.0448265171048474 _runtime‚3584.208688264 _step‚440 3loss/FBt1/assignment_loss‚0.0037663896605531592 /loss/FBt1/detection_loss‚0.6072504505569185 /loss/FBt2/detection_loss‚0.5342087036569287—šÐʼn epoch‚13 0loss/FRt1/assignment_loss‚0.9407352674980552 0loss/FRt2/assignment_loss‚2.0305763549538725 2loss/SRqqt1/assignment_loss‚0.6282078855631497 2loss/SRqqt2/assignment_loss‚0.7194649030670542 /loss/FRt1/detection_loss‚1.5626760176487495 /loss/FRt2/detection_loss‚0.5740782634176026 1loss/SRqqt1/detection_loss‚0.7149391920071968 1loss/FBt2/assignment_loss‚0.17510880404180815 %loss/total_loss‚9.535838749176737 1loss/SRqqt2/detection_loss‚1.0448265171048474 _runtime‚3584.208688264 _step‚440 3loss/FBt1/assignment_loss‚0.0037663896605531592 /loss/FBt1/detection_loss‚0.6072504505569185 /loss/FBt2/detection_loss‚0.5342087036569287 trainer/global_step‚5349 % _timestamp‚1.7844964276071002e+09¸³(5<Æ‚(:± ¬‚õÒɤï¾ disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚21386.5595703125$ disk.nvme1n1.out‚10351.076171875% disk./.usageGB‚1168.5884742736816 disk.md0.in‚0 disk.md0.out‚ 94.51953125 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ proc.memory.availableMB‚129.3125# proc.memory.rssMB‚ 2961.73046875 cpu‚25.812885490158514 network.sent‚ 7.839918e+06) disk./.usagePercent‚76.11714536874656' disk.nvme0c0n1.out‚21386.5595703125& disk.nvme1c1n1.out‚10351.076171875$ memory_percent‚99.21073913574219( proc.memory.percent‚18.0769681930542 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 network.recv‚969823 disk.dm-0.in‚0# disk.dm-0.out‚28958.97314453125" disk.dm-1.out‚2575.31787109375 disk.rbd0.in‚0Ã'‘ëžÇ‚(:“ ¬‚õÒ†ý’Á gpu.0.gpu‚52.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.733' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.24433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M>œ„EÈ‚(:º »‚õÒ‚åÒ¾ network.sent‚ 7.907029e+06 network.recv‚981105" disk.dm-1.out‚3078.05126953125 disk.nvme0c0n1.in‚0( disk.nvme1c1n1.out‚10854.19384765625 disk.nvme1n1.in‚0% disk./.usageGB‚1168.5885429382324 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 94.79296875$ memory_percent‚99.20792579650879 cpu‚25.812074383330938 proc.cpu.threads‚17) disk./.usagePercent‚76.11714984127877# disk.dm-0.out‚28959.45361328125 disk.rbd0.out‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚21387.25244140625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10854.19384765625' proc.memory.availableMB‚ 129.7734375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚21387.25244140625 disk.md0.in‚0# proc.memory.rssMB‚ 2961.73046875( proc.memory.percent‚18.0769681930542)Q†žÉ‚(:“ »‚õÒÀÄ¿ gpu.0.gpu‚91.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚227.093' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.69766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù&/>Ê‚(:³ Ê‚õÒ×èÞ¾ disk.rbd0.out‚0 disk.nvme0n1.in‚0# proc.memory.rssMB‚ 2961.73046875( proc.memory.percent‚18.0769681930542 network.sent‚ 7.911913e+06 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10855.13232421875 network.recv‚984516 disk.md0.out‚ 95.203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚21394.623046875$ disk.nvme0n1.out‚21394.623046875# memory_percent‚99.2079496383667 cpu‚25.811265284791915% disk./.usageGB‚1168.5889930725098" disk.dm-0.out‚28966.7998046875 disk.dm-1.in‚0" disk.dm-1.out‚3078.12158203125( proc.memory.availableMB‚ 129.76953125 proc.cpu.threads‚17) disk./.usagePercent‚76.11717916121216& disk.nvme1n1.out‚10855.13232421875 disk.dm-0.in‚0êËî‘Ë‚(:† Ê‚õÒœ¡¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.711' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.237 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á«S2Ì- _wandbruntime‚3616  _runtime‚3616Â>’<ÈÍ ' lr-AdamW/pg1‚0.00013604557097705928 ' lr-AdamW/pg2‚0.00013604557097705928 trainer/global_step‚5399 $ _timestamp‚1.784496459767539e+09 _runtime‚3616.369338397 _step‚441 ¾g“ÍÎÇ trainer/global_step‚5399 $ _timestamp‚1.784496459767539e+09 _runtime‚3616.369338397 _step‚441 ' lr-AdamW/pg1‚0.00013604557097705928 ' lr-AdamW/pg2‚0.00013604557097705928¹b–22Ï- _wandbruntime‚3616  _runtime‚3616ó“g0â'ÐÜO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423232844870144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4451037083648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12218613 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54307871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54297728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚418781702087168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40929819 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54297728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12218803 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4582612408320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4451125743616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423232739170816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚318215349364224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40929625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚318215572917760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4582741852160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3616.372047571 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42783973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42783973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413842329208208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418781702087168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 % _timestamp‚1.7844964597682755e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28711016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54307871 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4438908641982 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42783971 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423232739170816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418781719126528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚194 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313632831065600 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚5399 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97081699 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418781719126528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313632736955904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42783971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200PÙj(dÑáO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423232844870144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4451037083648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97091844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4438821855556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28711012 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4451125743616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4582741852160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3616.372047571 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42783973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42783973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚413842329208208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚418781702087168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 % _timestamp‚1.7844964597682755e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28711016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54307871 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4438908641982 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42783971 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423232739170816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418781719126528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚194 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313632831065600 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LÃ/oîƒDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚5399 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97081699 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418781719126528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313632736955904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42783971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423232844870144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4451037083648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12218613 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54307871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54297728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚418781702087168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40929819 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54297728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12218803 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4582612408320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4451125743616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423232739170816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚318215349364224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40929625 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚318215572917760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844964601816561e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚172 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54307548 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418856868792832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4451911466496 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97099148 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418856868792832 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97109307 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318272525746176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚3616.783625427 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42791600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40936826 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423308674552832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚413916636772960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4439692136694 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40936998 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚418856851753472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4583556952576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97109307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42791600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4451822799360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28716068 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12220926 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413916619733600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4439605350156 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313688968793600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423308780259328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423308780259328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97099148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54307548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54317705 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42791602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136J–÷ñ&ÔëM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313688968793600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423308780259328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423308780259328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97099148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54307548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54317705 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42791602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚418356328909654 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844964601816561e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚172 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54307548 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418856868792832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4451911466496 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97099148 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418856868792832 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97109307 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318272525746176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚3616.783625427 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42791600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40936826 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423308674552832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚413916636772960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4439692136694 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40936998 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚418856851753472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4583556952576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97109307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42791600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4451822799360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28716068 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12220926 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚413916619733600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4439605350156 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157»f«å2Õ- _wandbruntime‚3616  _runtime‚3616ªƒ¢IÖ€ 2loss/SRqqt1/assignment_loss‚0.5365424458511655 2loss/SRqqt2/assignment_loss‚0.6590380760783469 /loss/FRt2/detection_loss‚0.5332505939886961 1loss/SRqqt2/detection_loss‚0.9662380294945208 0loss/SRqqt1/detection_loss‚0.651749332761134 %loss/total_loss‚9.040742049910419 trainer/global_step‚5399 0loss/FRt1/assignment_loss‚0.7867906564859081 1loss/FBt1/assignment_loss‚0.03217963602477901 /loss/FRt1/detection_loss‚1.5804239410434109 _runtime‚3616.784159658 _step‚444 0loss/FBt2/assignment_loss‚0.0910782103043406 /loss/FBt1/detection_loss‚0.4328357242939603 /loss/FBt2/detection_loss‚0.7575496003652612 epoch‚13 % _timestamp‚1.7844964601826544e+09 0loss/FRt2/assignment_loss‚2.0130658032188955iåF3 ×… 0loss/FRt1/assignment_loss‚0.7867906564859081 1loss/FBt1/assignment_loss‚0.03217963602477901 /loss/FRt1/detection_loss‚1.5804239410434109 _runtime‚3616.784159658 _step‚444 0loss/FBt2/assignment_loss‚0.0910782103043406 /loss/FBt1/detection_loss‚0.4328357242939603 /loss/FBt2/detection_loss‚0.7575496003652612 epoch‚13 % _timestamp‚1.7844964601826544e+09 0loss/FRt2/assignment_loss‚2.0130658032188955 2loss/SRqqt1/assignment_loss‚0.5365424458511655 2loss/SRqqt2/assignment_loss‚0.6590380760783469 /loss/FRt2/detection_loss‚0.5332505939886961 1loss/SRqqt2/detection_loss‚0.9662380294945208 0loss/SRqqt1/detection_loss‚0.651749332761134 %loss/total_loss‚9.040742049910419 trainer/global_step‚5399¼_ÍíßBØ‚(:· Ù‚õÒ…ø¿ network.recv‚988402$ memory_percent‚99.20816421508789( proc.memory.percent‚18.0769681930542 disk.nvme0n1.in‚0& disk.nvme0n1.out‚21395.56005859375) disk./.usagePercent‚76.11709368615212 disk.dm-0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚21395.56005859375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10855.70263671875 network.sent‚ 7.917379e+06" disk.dm-0.out‚28967.5458984375 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 95.47265625& proc.memory.availableMB‚ 129.734375 proc.cpu.threads‚17% disk./.usageGB‚1168.5876808166504! disk.dm-1.out‚3078.2978515625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme1n1.out‚10855.70263671875# proc.memory.rssMB‚ 2961.73046875 cpu‚25.810389899749755ú`^œÙ‚(:‘ Ù‚õÒ´Þþ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.667' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.22233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šˆ®=Ú‚(:² è‚õÒ¾í¾$ disk./.usageGB‚1168.587760925293 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚10856.333984375 disk.md0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2961.73046875 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚21397.49267578125 disk.md0.out‚ 95.828125 disk.rbd0.out‚0# memory_percent‚99.2081880569458( proc.memory.percent‚18.0769681930542 cpu‚25.809531359497985 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚10856.333984375# disk.dm-0.out‚28969.26123046875! disk.dm-1.out‚3078.3759765625 disk.rbd0.in‚0 network.recv‚998923( disk.nvme0c0n1.out‚21397.49267578125 disk.dm-0.in‚0( proc.memory.availableMB‚ 129.73046875 proc.cpu.threads‚17 network.sent‚ 7.981695e+06) disk./.usagePercent‚76.11709890410636çÒۜۂ(:‘ è‚õÒßß«¿ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚162.229' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚54.07633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k(Å2Ü- _wandbruntime‚3648  _runtime‚3648*0G>ÉÝà % _timestamp‚1.7844964920005283e+09 _runtime‚3648.602368809 _step‚445 ' lr-AdamW/pg1‚0.00013558199550007855 ' lr-AdamW/pg2‚0.00013558199550007855 trainer/global_step‚5449\ð…ÎÞÈ ' lr-AdamW/pg1‚0.00013558199550007855 ' lr-AdamW/pg2‚0.00013558199550007855 trainer/global_step‚5449 % _timestamp‚1.7844964920005283e+09 _runtime‚3648.602368809 _step‚445½_™2ß- _wandbruntime‚3648  _runtime‚3648ÞÕ°¦ àÛO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43165423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _step‚446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229844992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54790580 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321062741044224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54800723 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚422539202441728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4490387023872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54800723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43165423 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚422034915864172 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97966146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28963812 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4478060386364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417556855477808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12324593 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316439623355904 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427029572426240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚422539185402368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54790580 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12324765 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚422539185402368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4490475683840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41288581 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 3648.60316309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0:½X¯æ'áàO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43165423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 _step‚446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229844992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54790580 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321062741044224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54800723 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚422539202441728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4490387023872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54800723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43165423 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚422034915864172 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97966146 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28963812 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4478060386364 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚417556855477808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12324593 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316439623355904 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427029572426240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚422539185402368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54790580 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12324765 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚422539185402368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4490475683840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41288581 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 3648.60316309 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0¾‹Â‘2â- _wandbruntime‚3649  _runtime‚3649Ú†êýì&ãæM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28968868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427105507808256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12326808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54810557 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844964924156435e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4491261406720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43173050 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54800400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422110093710062 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28968872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4624055678976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54800400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97973450 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54810557 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321119910872576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4623924145152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43173052 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41295858 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97973450 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4491172739584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417631163042560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43173050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427105613514752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚422614335068672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚422614335068672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97983609 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41295676 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4491172739584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 trainer/global_step‚5449 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97983609 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚422614352108032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚422614352108032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4478843880964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316495761083904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427105613514752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417631146003200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _step‚447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316495855193600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12326986 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚3649.017706478 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321119685229056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4491261406720 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175å¤:äëM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4491261406720 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28968868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427105507808256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12326808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54810557 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚178 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844964924156435e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4491261406720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43173050 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54800400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422110093710062 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28968872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4624055678976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54800400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97973450 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54810557 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321119910872576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4623924145152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43173052 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41295858 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97973450 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4491172739584 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417631163042560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 OADeviceStatsMonitor.on_train_batch_end/allocated>s3ð_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43173050 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427105613514752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚422614335068672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚422614335068672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97983609 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41295676 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4491172739584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 trainer/global_step‚5449 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97983609 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚422614352108032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚422614352108032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4478843880964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316495761083904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427105613514752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚417631146003200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 _step‚447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316495855193600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚182 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12326986 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 _runtime‚3649.017706478 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321119685229056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¿pn‹2å- _wandbruntime‚3649  _runtime‚3649bêøæ epoch‚13 0loss/FRt1/assignment_loss‚0.8584769139344665 2loss/SRqqt2/assignment_loss‚0.6731614690607354 1loss/FBt2/assignment_loss‚0.08086961760477861 1loss/SRqqt1/detection_loss‚0.6678618091614831 _runtime‚ 3649.01831447 _step‚448 /loss/FRt1/detection_loss‚1.5457583247694646 /loss/FRt2/detection_loss‚0.5149619481702372 /loss/FBt2/detection_loss‚1.2853541904523214 % _timestamp‚1.7844964924167655e+09 /loss/FRt2/assignment_loss‚2.015458115627026 2loss/FBt1/assignment_loss‚0.023103195251644528 1loss/SRqqt2/detection_loss‚1.0093635443022082 /loss/FBt1/detection_loss‚0.5020128403805597 2loss/SRqqt1/assignment_loss‚0.6158511585204911 %loss/total_loss‚9.792233127235416 trainer/global_step‚5449¤gOœ ç† 1loss/FBt2/assignment_loss‚0.08086961760477861 1loss/SRqqt1/detection_loss‚0.6678618091614831 _runtime‚ 3649.01831447 _step‚448 /loss/FRt1/detection_loss‚1.5457583247694646 /loss/FRt2/detection_loss‚0.5149619481702372 /loss/FBt2/detection_loss‚1.2853541904523214 % _timestamp‚1.7844964924167655e+09 /loss/FRt2/assignment_loss‚2.015458115627026 2loss/FBt1/assignment_loss‚0.023103195251644528 1loss/SRqqt2/detection_loss‚1.0093635443022082 /loss/FBt1/detection_loss‚0.5020128403805597 2loss/SRqqt1/assignment_loss‚0.6158511585204911 %loss/total_loss‚9.792233127235416 trainer/global_step‚5449 epoch‚13 0loss/FRt1/assignment_loss‚0.8584769139344665 2loss/SRqqt2/assignment_loss‚0.6731614690607354À_§ÍrJè‚(:¿ ÷‚õÒÛ²‚¿% disk./.usageGB‚1167.7723388671875 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 2961.73046875 network.sent‚ 8.00036e+06) disk./.usagePercent‚76.06398559629946 disk.nvme0n1.in‚0 disk.md0.out‚ 96.05859375# disk.dm-0.out‚28999.87060546875" disk.dm-1.out‚3078.57958984375( disk.nvme0c0n1.out‚21428.26318359375& disk.nvme0n1.out‚21428.26318359375& disk.nvme1n1.out‚10856.90185546875$ memory_percent‚99.20856952667236( proc.memory.availableMB‚ 129.66796875( proc.memory.percent‚18.0769681930542 cpu‚25.80860541737347 proc.cpu.threads‚17 network.recv‚ 1.002326e+06( disk.nvme1c1n1.out‚10856.90185546875 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0Ïê.Bé‚(:’ ÷‚õÒÿ©¶¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚165.674' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚55.224666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08hREê‚(:º †ƒõÒ€ýÞ¾ network.sent‚ 8.048786e+06 disk.dm-1.in‚0( disk.nvme1c1n1.out‚10858.43212890625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 96.32421875' proc.memory.availableMB‚ 129.3984375 network.recv‚ 1.006681e+06) disk./.usagePercent‚76.06397739665708% disk.nvme0n1.out‚21433.5126953125 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10858.43212890625 disk.dm-0.out‚29005.98046875% disk./.usageGB‚1167.7722129821777 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ memory_percent‚99.21021461486816# proc.memory.rssMB‚ 2961.73046875( proc.memory.percent‚18.0769681930542 cpu‚25.80776123828048 proc.cpu.threads‚17" disk.dm-1.out‚3078.64990234375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚21433.5126953125 disk.nvme0n1.in‚0 disk.dm-0.in‚0>„´–’ë‚(:‡ †ƒõÒï‹Ï¾ gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚220.947' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<]o­2ì- _wandbruntime‚3681  _runtime‚3681¢Ä/üÉíà _step‚449 ' lr-AdamW/pg1‚0.00013511486455436658 ' lr-AdamW/pg2‚0.00013511486455436658 trainer/global_step‚5499 % _timestamp‚1.7844965246064165e+09 _runtime‚3681.208241275µ`óbÎîÈ ' lr-AdamW/pg1‚0.00013511486455436658 ' lr-AdamW/pg2‚0.00013511486455436658 trainer/global_step‚5499 % _timestamp‚1.7844965246064165e+09 _runtime‚3681.208241275 _step‚449ÁH*öç2ï- _wandbruntime‚3681  _runtime‚3681a5jÄâ'ðÜO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421271398786768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29216612 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚450 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4529736964096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55293575 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4663666118656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚426296685756928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323910044333568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55283432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430826405681664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55283432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚167 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4529825624064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319246509755904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12430645 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425788680664580 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4517298917172 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319246603865600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚426296685756928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚426296668717568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4517385703598 % _timestamp‚1.7844965246074061e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98830303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98840448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425788784490366 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4529825624064 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430826405681664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41647428 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55293575 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4529736964096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29216616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421271381747408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323910269984256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43546873 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43546871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚430826511380992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4663534577664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚426296668717568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3681.209241497 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41647257 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248SÐ’oç'ñáO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323910269984256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43546873 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43546871 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚430826511380992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4663534577664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚426296668717568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3681.209241497 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41647257 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421271398786768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29216612 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚450 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4529736964096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55293575 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4663666118656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚426296685756928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323910044333568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55283432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430826405681664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55283432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚167 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4529825624064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319246509755904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12430645 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425788680664580 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131540992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4517298917172 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319246603865600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚426296685756928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚426296668717568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4517385703598 % _timestamp‚1.7844965246074061e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98830303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98840448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425788784490366 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4529825624064 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430826405681664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41647428 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55293575 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4529736964096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29216616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421271381747408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760·2ò- _wandbruntime‚3681  _runtime‚36812ªñ§ óåM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98857911 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98857911 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚426371835423232 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98847752 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430902446770176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3681.623009508 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚165 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12432992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319302647483904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421345689312160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 $ _timestamp‚1.784496525020969e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41654664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98847752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425863858510470 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55293252 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚5499 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55303409 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421345672272800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4530611346944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43554500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41654499 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43554502 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12432831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323967219404800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚426371818383872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43554502 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚451 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319302741593600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29221672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4518082411772 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29221668 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130®ºð&ôêM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4530611346944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43554500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41654499 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43554502 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12432831 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323967219404800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚426371818383872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43554502 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚451 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319302741593600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29221672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4518082411772 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29221668 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221449216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98857911 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98857911 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚426371835423232 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98847752 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430902446770176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚3681.623009508 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚165 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12432992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319302647483904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421345689312160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 $ _timestamp‚1.784496525020969e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41654664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98847752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425863858510470 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55293252 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 trainer/global_step‚5499 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55303409 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421345672272800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130ÃLÃo2õ- _wandbruntime‚3681  _runtime‚3681Ö On öƒ epoch‚13 0loss/FRt1/assignment_loss‚0.7678301715294014 2loss/SRqqt1/assignment_loss‚0.6110728424274167 2loss/SRqqt2/assignment_loss‚0.6803896578494459 /loss/FRt1/detection_loss‚1.4430325293514064 trainer/global_step‚5499 _step‚452 0loss/FRt2/assignment_loss‚1.9576927471583283 0loss/SRqqt2/detection_loss‚1.036913592344195 %loss/total_loss‚8.636051574690132 3loss/FBt1/assignment_loss‚0.0012486218321954886 1loss/FBt2/assignment_loss‚0.09071921844107612 0loss/SRqqt1/detection_loss‚0.596899760008377 0loss/FBt1/detection_loss‚0.44870338912309604 /loss/FRt2/detection_loss‚0.5399443777078622 /loss/FBt2/detection_loss‚0.4616046669173308 % _timestamp‚1.7844965250220141e+09 _runtime‚3681.623514939©½$÷ˆ %loss/total_loss‚8.636051574690132 3loss/FBt1/assignment_loss‚0.0012486218321954886 1loss/FBt2/assignment_loss‚0.09071921844107612 0loss/SRqqt1/detection_loss‚0.596899760008377 0loss/FBt1/detection_loss‚0.44870338912309604 /loss/FRt2/detection_loss‚0.5399443777078622 /loss/FBt2/detection_loss‚0.4616046669173308 % _timestamp‚1.7844965250220141e+09 _runtime‚3681.623514939 epoch‚13 0loss/FRt1/assignment_loss‚0.7678301715294014 2loss/SRqqt1/assignment_loss‚0.6110728424274167 2loss/SRqqt2/assignment_loss‚0.6803896578494459 /loss/FRt1/detection_loss‚1.4430325293514064 trainer/global_step‚5499 _step‚452 0loss/FRt2/assignment_loss‚1.9576927471583283 0loss/SRqqt2/detection_loss‚1.036913592344195ĉs4/Jø‚(:¿ •ƒõÒÒ†Ÿ¿( proc.memory.availableMB‚ 129.16796875( proc.memory.percent‚18.0769681930542 proc.cpu.threads‚17 network.sent‚ 8.067469e+06 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10859.24951171875 disk.md0.in‚0 disk.md0.out‚ 96.9453125# disk.dm-0.out‚29006.97509765625 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚21435.10205078125 disk.dm-0.in‚0$ memory_percent‚99.21162128448486) disk./.usagePercent‚76.06398037834522$ disk./.usageGB‚1167.772258758545( disk.nvme1c1n1.out‚10859.24951171875 disk.dm-1.in‚0" disk.dm-1.out‚3078.74755859375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2961.73046875 cpu‚25.806777553570626 network.recv‚ 1.010361e+06 disk.nvme0n1.in‚0& disk.nvme0n1.out‚21435.10205078125 disk.nvme1c1n1.in‚0 disk.rbd0.out‚03užù‚(:“ •ƒõÒÊÛÑ¿ gpu.0.gpu‚53.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚177.728' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚59.242666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©©¢nAú‚(:¶ ¤ƒõÒÂÓô¾ network.sent‚ 8.115948e+06" disk.dm-0.out‚29014.4052734375 disk.dm-1.out‚3078.873046875 disk.rbd0.in‚0 disk.nvme0n1.in‚0# memory_percent‚99.2112398147583( proc.memory.availableMB‚ 129.23046875 cpu‚25.805948327714823 disk.md0.out‚ 97.21484375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚21442.369140625$ disk.nvme0n1.out‚21442.369140625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2961.73046875( proc.memory.percent‚18.0769681930542 proc.cpu.threads‚17( disk./.usagePercent‚76.0660541424474% disk.nvme1n1.out‚10860.1337890625 disk.md0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚10860.1337890625 network.recv‚ 1.014738e+06% disk./.usageGB‚1167.8040962219238 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0 ²˜0“û‚(:ˆ ¤ƒõÒÌð°¿ gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚114.861' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.287 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·©l2ü- _wandbruntime‚3713  _runtime‚3713T ÔÉýà trainer/global_step‚5549 % _timestamp‚1.7844965568404698e+09 _runtime‚3713.442364033 _step‚453 ' lr-AdamW/pg1‚0.00013464420814287114 ' lr-AdamW/pg2‚0.00013464420814287114k)qÎþÈ % _timestamp‚1.7844965568404698e+09 _runtime‚3713.442364033 _step‚453 ' lr-AdamW/pg1‚0.00013464420814287114 ' lr-AdamW/pg2‚0.00013464420814287114 trainer/global_step‚5549ÅÃÞ‰&2ÿ- _wandbruntime‚3713  _runtime‚3713i-‚Fâ'€ÜO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4569086904320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43928323 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29469412 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29469416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434623238937088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43928321 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12538065 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12538249 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43928323 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚430054152032768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43928321 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4704009488896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚424985908017008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135735296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _runtime‚3713.443277575 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42007477 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429542549290774 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4569175564288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326757635489792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4569086904320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55786427 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429542445464988 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55776284 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚434623344636416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434623344636416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322053396155904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚454 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430054169072128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434623238937088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99714750 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229844992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4704145224192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322053490265600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4556537447980 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99704605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430054152032768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4569175564288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326757405644800b¦R¬áO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99704605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚430054152032768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4569175564288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326757405644800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4569086904320 TGDeviceStatsMonitor.on_train_bat~Û‡•f%ch_start/allocation.large_pool.allocated‚43928323 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29469412 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29469416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434623238937088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43928321 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12538065 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12538249 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43928323 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚430054152032768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43928321 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4704009488896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚424985908017008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135735296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚188 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _runtime‚3713.443277575 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42007477 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429542549290774 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4569175564288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326757635489792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4569086904320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55786427 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429542445464988 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55776284 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚434623344636416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434623344636416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322053396155904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 _step‚454 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚430054169072128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434623238937088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99714750 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229844992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4704145224192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322053490265600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4556537447980Æ\ì‰g2‚- _wandbruntime‚3714  _runtime‚3714ï.F¤ë&ƒåM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4557407729118 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99722054 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚430129301699072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4704815552512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434699280025600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55796261 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55786104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434699280025600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29474472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12540367 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55786104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434699174319104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4704944989184 _runtime‚3714.156314316 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42014661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326814349436416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42014839 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99732213 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43935952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429617623310878 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4569872620032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4569961287168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784496557554332e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430129318738432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425060215581760 _step‚455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43935950 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29474468 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4569961287168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99732213 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚430129301699072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4569872620032ã¶ð&„êM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42014839 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99732213 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43935952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429617623310878 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4569872620032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4569961287168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784496557554332e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚430129318738432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425060215581760 _step‚455 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43935950 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29474468 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4569961287168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99732213 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚430129301699072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4569872620032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4557407729118 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99722054 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚430129301699072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4704815552512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434699280025600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 129436672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55796261 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55786104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434699280025600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29474472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12540367 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55786104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434699174319104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4704944989184 _runtime‚3714.156314316 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚178 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42014661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326814349436416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5549 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760ǧ-ã}2…- _wandbruntime‚3714  _runtime‚3714’T–#†€ /loss/FBt2/detection_loss‚0.9335145011397177 %loss/total_loss‚9.506169620672738 /loss/FRt2/detection_loss‚0.5705227126753499 0loss/SRqqt1/detection_loss‚0.630179741040128 1loss/SRqqt2/detection_loss‚1.1024165997238065 trainer/global_step‚5549 % _timestamp‚1.7844965575553715e+09 _runtime‚3714.156892857 /loss/FRt1/detection_loss‚1.4310526473565321 epoch‚13 _step‚456 0loss/FRt2/assignment_loss‚1.9893443803312314 2loss/SRqqt1/assignment_loss‚0.6448903279511831 1loss/FBt1/assignment_loss‚0.04965789335476926 0loss/FBt2/assignment_loss‚0.1108125616707247 /loss/FBt1/detection_loss‚0.6699124524957832 0loss/FRt1/assignment_loss‚0.7658603437323319 2loss/SRqqt2/assignment_loss‚0.6080054592011795è¨Å÷ ‡… /loss/FBt2/detection_loss‚0.9335145011397177 %loss/total_loss‚9.506169620672738 /loss/FRt2/detection_loss‚0.5705227126753499 0loss/SRqqt1/detection_loss‚0.630179741040128 1loss/SRqqt2/detection_loss‚1.1024165997238065 trainer/global_step‚5549 % _timestamp‚1.7844965575553715e+09 _runtime‚3714.156892857 /loss/FRt1/detection_loss‚1.4310526473565321 epoch‚13 _step‚456 0loss/FRt2/assignment_loss‚1.9893443803312314 2loss/SRqqt1/assignment_loss‚0.6448903279511831 1loss/FBt1/assignment_loss‚0.04965789335476926 0loss/FBt2/assignment_loss‚0.1108125616707247 /loss/FBt1/detection_loss‚0.6699124524957832 0loss/FRt1/assignment_loss‚0.7658603437323319 2loss/SRqqt2/assignment_loss‚0.6080054592011795Èš¬)¡?ˆ‚(:´ ³ƒõÒ¡´µ¿( proc.memory.percent‚18.0769681930542 disk.nvme0n1.in‚0% disk.nvme0n1.out‚21443.4892578125 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚29015.23046875 disk.dm-1.out‚3079.001953125 disk.rbd0.out‚0 network.recv‚ 1.018257e+06' disk.nvme0c0n1.out‚21443.4892578125 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚10860.7998046875 cpu‚25.805046709118592 proc.cpu.threads‚17 network.sent‚ 8.134671e+06 disk.nvme0c0n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 97.59765625 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.21150207519531) disk./.usagePercent‚76.06607799595254% disk./.usageGB‚1167.8044624328613' disk.nvme1c1n1.out‚10860.7998046875$ proc.memory.availableMB‚129.1875# proc.memory.rssMB‚ 2961.73046875點‰‚(:“ ³ƒõÒÏ•ã¿ gpu.0.gpu‚91.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.086' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.02866666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X—,—'Š‚(:º ƒõÒ£¨Š¿ disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚21445.931640625( disk.nvme1c1n1.out‚10861.97705078125 disk.md0.in‚0 disk.md0.out‚ 98.0546875( proc.memory.percent‚18.0769681930542) disk./.usagePercent‚76.06427755326328 disk.dm-1.o±6Q†ut‚3079.134765625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0( proc.memory.availableMB‚ 129.18359375% disk./.usageGB‚1167.7768211364746$ disk.nvme0n1.out‚21445.931640625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10861.97705078125$ memory_percent‚99.21152591705322 cpu‚25.804231170586707 network.sent‚ 8.182126e+06 network.recv‚ 1.021437e+06# disk.dm-0.out‚29017.75341796875# proc.memory.rssMB‚ 2961.73046875 proc.cpu.threads‚17”­úב‹‚(:† ƒõÒۆͿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.945' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.315 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h±2Œ- _wandbruntime‚3745  _runtime‚3745"ã¼æÉà _step‚457 ' lr-AdamW/pg1‚0.00013417005649497403 ' lr-AdamW/pg2‚0.00013417005649497403 trainer/global_step‚5599 % _timestamp‚1.7844965893638215e+09 _runtime‚3745.965634577m´ìøÎŽÈ % _timestamp‚1.7844965893638215e+09 _runtime‚3745.965634577 _step‚457 ' lr-AdamW/pg1‚0.00013417005649497403 ' lr-AdamW/pg2‚0.00013417005649497403 trainer/global_step‚5599Én`ñû2- _wandbruntime‚3745  _runtime‚3745칎~æ'àO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433811635347968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329604994412032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438420177891840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4744488302592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100578907 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _runtime‚3745.966637049 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433811652387328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433296314091182 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744617746432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324860282555904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4595862765214 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4608436844544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56269136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56269136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12644097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438420072192512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433811635347968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚438420072192512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42366309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844965893648105e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44309773 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4595775978788 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4608525504512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12644269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚5599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44309773 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433811652387328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚458”è Zë'‘åO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4595775978788 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4608525504512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12644269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 trainer/global_step‚5599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44309773 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433811652387328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚458 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433811635347968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329604994412032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438420177891840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4744488302592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100578907 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _runtime‚3745.966637049 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433811652387328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433296314091182 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744617746432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324860282555904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4595862765214 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4608436844544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56269136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56269136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚172 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12644097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438420072192512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433811635347968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚438420072192512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42366309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844965893648105e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44309773ÊáT¢+2’- _wandbruntime‚3746  _runtime‚3746Ðõ¿ð&“êM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433371284285388 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56278956 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4609222560256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4609311227392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56278956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29727272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329661720548352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100606515 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚459 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56289113 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚3746.374683094 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433886785014272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428774741851360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚438496113281024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44317402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428774724812000 % _timestamp‚1.7844965897726886e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44317402 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324916420283904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42373453 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100596356 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329661941997568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433371388111286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100596356 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4596559473388 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12646355 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433886802053632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44317400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4745300264448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44317400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433886785014272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4596646259926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324916514393600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚438496113281024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4745427603968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100606515 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚3746.374683094 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/actšÀMœß#ive_bytes.large_pool.freed‚433886785014272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚174 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 127339520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428774741851360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚438496113281024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44317402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚170 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428774724812000 % _timestamp‚1.7844965897726886e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44317402 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324916420283904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42373453 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100596356 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329661941997568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433371388111286 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100596356 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4596559473388 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12646355 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433886802053632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44317400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4745300264448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44317400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433886785014272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4596646259926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324916514393600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚438496113281024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4745427603968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100606515 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433371284285388 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56278956 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4609222560256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4609311227392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56278956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29727272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329661720548352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100606515 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚459Ë•È12•- _wandbruntime‚3746  _runtime‚3746PÖ&–€ 0loss/FRt2/assignment_loss‚2.0551112057863796 2loss/SRqqt1/assignment_loss‚0.6239842574227432 1loss/SRqqt2/assignment_loss‚0.688432208779074 1loss/FBt2/assignment_loss‚0.04557280490685874 /loss/FRt2/detection_loss‚0.5351302215303246 /loss/FRt1/detection_loss‚1.4424939865301802 /loss/FBt2/detection_loss‚0.6756236391165891 %loss/total_loss‚9.145619876919078 # _timestamp‚1.78449658977452e+09 1loss/SRqqt1/detection_loss‚0.6408063608524325 epoch‚13 trainer/global_step‚5599 _step‚460 0loss/FRt1/assignment_loss‚0.8240309235545507 3loss/FBt1/assignment_loss‚0.0053399186444022485 0loss/SRqqt2/detection_loss‚1.000682988242091 /loss/FBt1/detection_loss‚0.6084113615534518 _runtime‚3746.376220977-8†Ú —… /loss/FRt1/detection_loss‚1.4424939865301802 /loss/FBt2/detection_loss‚0.6756236391165891 %loss/total_loss‚9.145619876919078 # _timestamp‚1.78449658977452e+09 1loss/SRqqt1/detection_loss‚0.6408063608524325 epoch‚13 trainer/global_step‚5599 _step‚460 0loss/FRt1/assignment_loss‚0.8240309235545507 3loss/FBt1/assignment_loss‚0.0053399186444022485 0loss/SRqqt2/detection_loss‚1.000682988242091 /loss/FBt1/detection_loss‚0.6084113615534518 _runtime‚3746.376220977 0loss/FRt2/assignment_loss‚2.0551112057863796 2loss/SRqqt1/assignment_loss‚0.6239842574227432 1loss/SRqqt2/assignment_loss‚0.688432208779074 1loss/FBt2/assignment_loss‚0.04557280490685874 /loss/FRt2/detection_loss‚0.5351302215303246ÌسB8˜‚(:­ уõÒ¶êؾ network.recv‚ 1.025235e+06) disk./.usagePercent‚76.04879041658398 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10862.40869140625 disk.md0.in‚0 disk.md0.out‚ 98.2734375' proc.memory.availableMB‚ 127.1171875 network.sent‚ 8.187955e+06! disk.nvme0c0n1.out‚ 21446.6875 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 21446.6875 disk.dm-0.in‚0# memory_percent‚99.2241382598877# proc.memory.rssMB‚ 2963.73046875 cpu‚25.80634529631164% disk./.usageGB‚1167.5390548706055 disk.dm-1.in‚0 disk.dm-1.out‚3079.248046875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0( proc.memory.percent‚18.0891752243042 proc.cpu.threads‚17 disk.rbd0.in‚0( disk.nvme1c1n1.out‚10862.40869140625# disk.dm-0.out‚29018.34716796875zU!€‘™‚(:† уõÒ¥²´¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚113.955' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.985 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ¤à2š- _wandbruntime‚3759  _runtime‚3759 +}ð=›ê{ 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7897289991378784 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9271023869514465 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.863556981086731 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8329554200172424 particle/accuracy_1_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5074999928474426 _runtime‚3759.569472753 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926103949546814 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940080046653748 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20258167386054993 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5844900608062744 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6491662859916687 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7582852244377136 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.192963644862175 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1905873566865921 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3785431385040283 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8397580981254578 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6833016872406006 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9128988981246948 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2507539987564087 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5869794487953186 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8557844758033752 jet/accuracy_3_of_5‚1 _step‚461 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042202092707157135 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05558793991804123 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2371540516614914 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6917385458946228 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959702491760254 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6476292610168457 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8907423615455627 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.722819447517395 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24970830976963043 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.616867184638977 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8358277082443237 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5640050172805786 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19886252284049988 +jet/accuracy_3_of_3‚0.07257755845785141 /particle/accuracy_3_of_4‚0.9942177534103394 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2506534159183502 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12495835870504379 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/specificity‚0.9096929430961609 % _timestamp‚1.7844966029672103e+09 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.047599080950021744 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7305575013160706 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7947786450386047 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6312856078147888 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28334179520606995 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6878207921981812 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7291216254234314 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8408918976783752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.047090958803892136 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04760059341788292 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38519763946533203 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15036889910697937 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21623393893241882 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0007558579673059285 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_4‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997212827205658 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6752838492393494 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007558579673059285 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8672198057174683 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7367039918899536 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18072393536567688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6445214152336121 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6544274687767029 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4441068768501282 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8269023895263672 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964998960494995 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7590325474739075 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21430279314517975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5956389307975769 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8315192461013794 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6461480259895325 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19714662432670593 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7412514686584473 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38311541080474854 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2008647918701172 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.568281352519989 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3854166865348816 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7641478180885315 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32813286781311035 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_1_of_3‚0.9262919425964355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923388361930847 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4398832321166992 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 *particle/sensitivity‚0.630409300327301 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938425421714783 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6524404883384705 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 trainer/global_step‚5599 )loss/val_total_loss‚8.837165020799121 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9873892664909363 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6662089228630066 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23983266949653625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21062414348125458 +jet/accuracy_1_of_1‚0.33815473318099976 'particle/f_score‚0.6430615186691284 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4552949368953705 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5308808088302612 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23983266949653625 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2925565540790558 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 epoch‚13 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6524404883384705 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18354520201683044 *jet/accuracy_2_of_3‚0.5575149655342102 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5241614580154419 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950034618377686 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23832252621650696 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23128657042980194 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04791655018925667 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029191603884100914 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14020340144634247 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29915332794189453 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.969977617263794 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6968361735343933 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9819521903991699 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22027648985385895 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.452847421169281 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5875065922737122 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9489167332649231 particle/accuracy_5_of_5‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.984988808631897 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4407441318035126 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8557844758033752 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920517802238464 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13460101187229156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20492546260356903 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5906424522399902 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02143673412501812 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.668323814868927 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0477256141602993 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.040636807680130005 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04760059341788292 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9838445782661438 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.734145998954773 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4631819427013397 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6590667963027954 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7346304655075073 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7360411882400513 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7384390830993652 'particle/accuracy‚0.849717915058136 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9764949083328247 particle/accuracy_2_of_5‚1 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.792344331741333 +jet/accuracy_2_of_2‚0.14825351536273956 +jet/accuracy_3_of_4‚0.09062311053276062 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.989403486251831 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03148261830210686§kñˆ´œï{ 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6968361735343933 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9819521903991699 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22027648985385895 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.452847421169281 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5875065922737122 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9489167332649231 particle/accuracy_5_of_5‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.984988808631897 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4407441318035126 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8557844758033752 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920517802238464 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13460101187229156 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20492546260356903 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5906424522399902 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02143673412501812 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.668323814868927 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0477256141602993 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.040636807680130005 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04760059341788292 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9838445782661438 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.734145998954773 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4631819427013397 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6590667963027954 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7346304655075073 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7360411882400513 particle/accuracy_2_of_2‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7384390830993652 'particle/accuracy‚0.849717915058136 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9764949083328247 particle/accuracy_2_of_5‚1 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.792344331741333 +jet/accuracy_2_of_2‚0.14825351536273956 +jet/accuracy_3_of_4‚0.09062311053276062 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.989403486251831 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03148261830210686 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7897289991378784 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9271023869514465 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.863556981086731 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8329554200172424 particle/accuracy_1_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5074999928474426 _runtime‚3759.569472753 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926103949546814 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940080046653748 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20258167386054993 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5844900608062744 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6491662859916687 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7582852244377136 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.192963644862175 9"Purity/1FRt*SRqqt*FBt/event_purity‚õ5­ÐA+0.1905873566865921 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3785431385040283 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8397580981254578 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6833016872406006 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9128988981246948 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2507539987564087 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5869794487953186 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8557844758033752 jet/accuracy_3_of_5‚1 _step‚461 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042202092707157135 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05558793991804123 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2371540516614914 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6917385458946228 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959702491760254 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6476292610168457 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8907423615455627 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.722819447517395 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24970830976963043 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.616867184638977 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8358277082443237 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5640050172805786 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19886252284049988 +jet/accuracy_3_of_3‚0.07257755845785141 /particle/accuracy_3_of_4‚0.9942177534103394 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2506534159183502 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12495835870504379 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/specificity‚0.9096929430961609 % _timestamp‚1.7844966029672103e+09 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.047599080950021744 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7305575013160706 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7947786450386047 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6312856078147888 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28334179520606995 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6878207921981812 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7291216254234314 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8408918976783752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.047090958803892136 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04760059341788292 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38519763946533203 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15036889910697937 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21623393893241882 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0007558579673059285 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_4‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997212827205658 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6752838492393494 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007558579673059285 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8672198057174683 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7367039918899536 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18072393536567688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6445214152336121 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6544274687767029 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4441068768501282 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8269023895263672 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964998960494995 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7590325474739075 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21430279314517975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5956389307975769 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8315192461013794 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6461480259895325 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19714662432670593 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7412514686584473 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38311541080474854 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2008647918701172 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.568281352519989 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3854166865348816 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7641478180885315 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32813286781311035 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_1_of_3‚0.9262919425964355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923388361930847 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4398832321166992 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 *particle/sensitivity‚0.630409300327301 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938425421714783 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6524404883384705 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 trainer/global_step‚5599 )loss/val_total_loss‚8.837165020799121 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9873892664909363 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6662089228630066 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23983266949653625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21062414348125458 +jet/accuracy_1_of_1‚0.33815473318099976 'particle/f_score‚0.6430615186691284 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4552949368953705 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5308808088302612 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23983266949653625 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2925565540790558 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 epoch‚13 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6524404883384705 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18354520201683044 *jet/accuracy_2_of_3‚0.5575149655342102 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5241614580154419 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950034618377686 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23832252621650696 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23128657042980194 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04791655018925667 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.029191603884100914 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14020340144634247 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29915332794189453 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.969977617263794ÍqÍf¶J‚(:¿ àƒõÒï÷Ö¾ proc.cpu.threads‚17 disk.dm-0.in‚0$ disk./.usageGB‚1167.539134979248' disk.nvme1c1n1.out‚10927.1650390625% disk.nvme1n1.out‚10927.1650390625 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0) disk./.usagePercent‚76.04879563453822 disk.dm-1.in‚0! disk.dm-1.out‚3143.2158203125( disk.nvme0c0n1.out‚21451.31396484375 network.recv‚ 1.029439e+06 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 98.87109375 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 192.63671875# proc.memory.rssMB‚ 2971.65234375 cpu‚25.813597574816175 network.sent‚ 8.250203e+06# disk.dm-0.out‚29022.51318359375& disk.nvme0n1.out‚21451.31396484375$ memory_percent‚98.82423877716064* proc.memory.percent‚18.137526512145996î$gz“ž‚(:ˆ àƒõÒ€”´¿ gpu.0.gpu‚78.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚116.151' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.717 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@j^HŸ‚(:½ ïƒõÒ®¬¿) disk./.usagePercent‚76.07818687996789 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21624.60107421875' disk.nvme1c1n1.out‚11041.5693359375 proc.cpu.threads‚17 network.recv‚ 1.033898e+06 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚21624.60107421875% disk.nvme1n1.out‚11041.5693359375$ memory_percent‚98.82426261901855' proc.memory.availableMB‚ 192.6328125 network.sent‚ 8.308248e+06$ disk./.usageGB‚1167.990364074707 disk.md0.in‚0 disk.md0.out‚ 99.1171875# disk.dm-0.out‚29297.16552734375 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0! disk.dm-1.out‚3155.7041015625 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2971.65234375* proc.memory.percent‚18.137526512145996 cpu‚25.813204801483643²s埠‚(:” ïƒõÒ…úí¾ gpu.0.gpu‚79.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.809' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.269666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}‚Õq2¡- _wandbruntime‚3791  _runtime‚3791ž›«Oɢà ' lr-AdamW/pg1‚0.00013369244006454959 ' lr-AdamW/pg2‚0.00013369244006454959 trainer/global_step‚5649 % _timestamp‚1.7844966347575972e+09 _runtime‚3791.359428471 _step‚462·Ím\Î£È _step‚462 ' lr-AdamW/pg1‚0.00013369244006454959 ' lr-AdamW/pg2‚0.00013369244006454959 trainer/global_step‚5649 % _timestamp‚1.7844966347575972e+09 _runtime‚3791.359428471Îá~®2¤- _wandbruntime‚3791  _runtime‚3791×úL‘å'¥ßO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443228822120960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328470628270592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443228927820288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438565622998528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333271214164992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4663216161792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚328470534160896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4650429768162 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4663304821760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813023 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56844495 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚438038961186348 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101667661 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56854638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30083267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4663216161792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333271437718528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101657516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4800680004096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4650516554588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12798753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30083271 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 $ _timestamp‚1.784496634758563e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚438565622998528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438038857360562 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443228822120960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4663304821760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813021 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813023 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56854638 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143,?TÎ ¦äO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101657516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4800680004096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4650516554588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12798753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30083271 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 $ _timestamp‚1.784496634758563e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚438565622998528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438038857360562 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443228822120960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4663304821760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813021 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813023 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56854638 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443228822120960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328470628270592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443228927820288 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚438565622998528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333271214164992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4663216161792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚328470534160896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4650429768162 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4663304821760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813023 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56844495 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚438038961186348 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101667661 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56854638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30083267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4663216161792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333271437718528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ï•lçä2§- _wandbruntime‚3791  _runtime‚3791Ù?r·ð&¨êM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚438113931380554 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44820652 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4664001877504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12800710 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328526671888896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333328378856960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3791.760658768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56864472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443304863209472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44820650 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56864472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433462735157152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44820652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30088323 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44820650 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438640755625472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4651213262762 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101685124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4801493907456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438640755625472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56854315 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4664090544640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101674965 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4801612858368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4664001877504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433462718117792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42889033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30088327 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101674965 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438640772664832 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443304757502976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚5649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42889224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333328165796352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101685124 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682Ž|cÍõ&©ïM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4664001877504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12800710 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 _step‚464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328526671888896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333328378856960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3791.760658768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56864472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443304863209472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44820650 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56864472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433462735157152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44820652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30088323 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44820650 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚438640755625472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4651213262762 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101685124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4801493907456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚438640755625472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56854315 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4664090544640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101674965 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4801612858368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4664001877504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433462718117792 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42889033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30088327 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101674965 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚438640772664832 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443304757502976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 trainer/global_step‚5649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42889224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333328165796352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101685124 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚438113931380554 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44820652 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213060608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360И«š2ª- _wandbruntime‚3791  _runtime‚3791NXYÌ«€ trainer/global_step‚5649 1loss/SRqqt1/assignment_loss‚0.583498953526939 1loss/SRqqt2/assignment_loss‚0.680345688249202 2loss/FBt2/assignment_loss‚0.026872967768164814 /loss/FRt1/detection_loss‚1.5807874372801165 /loss/FRt2/detection_loss‚0.5586584649772667 _runtime‚ 3791.76122841 _step‚465 0loss/FRt1/assignment_loss‚0.7811592771856354 1loss/SRqqt1/detection_loss‚0.6130614542725933 %loss/total_loss‚9.023765139719295 % _timestamp‚1.7844966351595974e+09 0loss/FRt2/assignment_loss‚2.0671299087201676 1loss/SRqqt2/detection_loss‚0.9457452015307545 epoch‚14 1loss/FBt1/assignment_loss‚0.00244303381169635 /loss/FBt1/detection_loss‚0.3935901105068209 /loss/FBt2/detection_loss‚0.7904726418899373˜Ç§º ¬… 1loss/FBt1/assignment_loss‚0.00244303381169635 /loss/FBt1/detection_loss‚0.3935901105068209 /loss/FBt2/detection_loss‚0.7904726418899373 trainer/global_step‚5649 1loss/SRqqt1/assignment_loss‚0.583498953526939 1loss/SRqqt2/assignment_loss‚0.680345688249202 2loss/FBt2/assignment_loss‚0.026872967768164814 /loss/FRt1/detection_loss‚1.5807874372801165 /loss/FRt2/detection_loss‚0.5586584649772667 _runtime‚ 3791.76122841 _step‚465 0loss/FRt1/assignment_loss‚0.7811592771856354 1loss/SRqqt1/detection_loss‚0.6130614542725933 %loss/total_loss‚9.023765139719295 % _timestamp‚1.7844966351595974e+09 0loss/FRt2/assignment_loss‚2.0671299087201676 1loss/SRqqt2/detection_loss‚0.9457452015307545 epoch‚14ÑtvöÚF­‚(:» þƒõÒÛÑä¾ disk.dm-0.in‚0& disk.nvme0c0n1.out‚22840.693359375& disk.nvme1n1.out‚11267.30712890625 disk.md0.out‚ 99.58984375* proc.memory.percent‚18.137526512145996 proc.cpu.threads‚17! disk.dm-0.out‚30685.693359375 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚22840.693359375 disk.md0.in‚0' proc.memory.availableMB‚ 192.6484375 cpu‚25.81440787140341% disk./.usageGB‚1169.5182189941406 disk.dm-1.in‚0" disk.dm-1.out‚3208.01025390625( disk.nvme1c1n1.out‚11267.30712890625 disk.nvme1n1.in‚0 network.sent‚ 8.311212e+06 network.recv‚ 1.037181e+06) disk./.usagePercent‚76.17770519421205 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚98.82416725158691# proc.memory.rssMB‚ 2971.65234375«ƒ^Î’®‚(:‡ þƒõÒ¢ÒÚ¿ gpu.0.gpu‚15.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚200.328' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.776 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô¯H¯‚(:½ „õÒÕ“ý¾) disk./.usagePercent‚76.18177470157674 disk.nvme1n1.in‚0! disk.dm-0.out‚31057.392578125* proc.memory.percent‚18.137526512145996( disk.nvme1c1n1.out‚11268.06982421875 disk.md0.in‚0" disk.dm-1.out‚3208.23291015625 disk.rbd0.out‚0$ memory_percent‚98.82595539093018 cpu‚25.8163099687675 network.sent‚ 8.375569e+06 disk.nvme0n1.in‚0& disk.nvme1n1.out‚11268.06982421875 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 2971.65234375$ disk./.usageGB‚1169.580696105957% disk.nvme0n1.out‚23212.7841796875 disk.nvme1c1n1.in‚0 disk.md0.out‚ 100.02734375' disk.nvme0c0n1.out‚23212.7841796875( proc.memory.availableMB‚ 192.35546875 proc.cpu.threads‚17 network.recv‚ 1.041694e+06 %1I°‚(:’ „õÒŠ™¬¿ gpu.0.gpu‚53.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚223.853' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.61766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒDt§2±- _wandbruntime‚3823  _runtime‚3823¯ù´É²Ã ' lr-AdamW/pg1‚0.00013321138952800864 ' lr-AdamW/pg2‚0.00013321138952800864 trainer/global_step‚5699 % _timestamp‚1.7844966664696467e+09 _runtime‚3823.072259553 _step‚466md³Î³È ' lr-AdamW/pg1‚0.00013321138952800864 ' lr-AdamW/pg2‚0.00013321138952800864 trainer/global_step‚5699 % _timestamp‚1.7844966664696467e+09 _runtime‚3823.072259553 _step‚466ÒÝßx2´- _wandbruntime‚3823  _runtime‚3823+‰žObµàO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4702566102016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚201 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚447025761075712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331277514670592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57337347 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 125249536 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447025655376384 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4841316692480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30336067 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45194473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442323089274368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57347490 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442323089274368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441792725986756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4689668298970 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57337347 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45194471 KADeviceSt<Á„atsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚5699 % _timestamp‚1.7844966664703722e+09 _runtime‚3823.072349853 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4702566102016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45194471 _step‚467 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚437102953862000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331277420560896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447025655376384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442323106313728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102531818 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441792622160970 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4702654761984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102541963 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43244712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚197 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336118737253376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447025655376384 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4841316692480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30336067 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45194473 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442323089274368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57347490 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442323089274368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441792725986756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4689668298970 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57337347 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45194471 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 trainer/global_step‚5699 % _timestamp‚1.7844966664703722e+09 _runtime‚3823.072349853 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4702566102016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45194471 _step‚467 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚437102953862000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331277420560896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚447025655376384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442323106313728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ÓkªF22·- _wandbruntime‚3823  _runtime‚3823C¸»pï&¸éM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12910600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4690538580108 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4690451793570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 $ _timestamp‚1.784496666881237e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442398255980032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4703351817728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57357324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102559426 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102549267 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447101590758400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102549267 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4842249415680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331333652398592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57347167 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441867800006860 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441867696180962 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4842130464768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43251723 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336175688753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437177261426752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚468 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚3823.483229528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4703440484864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12910779 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102559426 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4703351817728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703440484864s M!¹îM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4842130464768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43251723 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336175688753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437177261426752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚468 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 _runtime‚3823.483229528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4703440484864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12910779 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118950912 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102559426 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4703351817728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703440484864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12910600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4690538580108 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4690451793570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 $ _timestamp‚1.784496666881237e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442398255980032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4703351817728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57357324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102559426 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 E=DeviceStatsMonitor.*ìΧon_train_batch_end/segment.large_pool.peak‚682 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102549267 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447101590758400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102549267 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4842249415680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331333652398592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57347167 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441867800006860 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441867696180962 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ôé^ L2º- _wandbruntime‚3823  _runtime‚3823Ec©·»‚ 2loss/SRqqt1/assignment_loss‚0.5274460347989345 2loss/FBt1/assignment_loss‚0.039065108404556806 /loss/FRt1/detection_loss‚1.4847290368861836 0loss/SRqqt1/detection_loss‚0.639338817942189 % _timestamp‚1.7844966668822641e+09 0loss/FRt1/assignment_loss‚0.7790343774934855 0loss/FRt2/assignment_loss‚2.0505200273850384 1loss/FBt2/assignment_loss‚0.11250244549862669 %loss/total_loss‚9.309712948748972 epoch‚14 trainer/global_step‚5699 _runtime‚3823.483815439 2loss/SRqqt2/assignment_loss‚0.6603520061359635 1loss/SRqqt2/detection_loss‚0.9651704413895207 /loss/FBt1/detection_loss‚0.5601308784923394 /loss/FBt2/detection_loss‚0.9560224630703933 /loss/FRt2/detection_loss‚0.5354013112517398 _step‚469pÓµO ¼‡ 1loss/SRqqt2/detection_loss‚0.9651704413895207 /loss/FBt1/detection_loss‚0.5601308784923394 /loss/FBt2/detection_loss‚0.9560224630703933 /loss/FRt2/detection_loss‚0.5354013112517398 _step‚469 2loss/SRqqt1/assignment_loss‚0.5274460347989345 2loss/FBt1/assignment_loss‚0.039065108404556806 /loss/FRt1/detection_loss‚1.4847290368861836 0loss/SRqqt1/detection_loss‚0.639338817942189 % _timestamp‚1.7844966668822641e+09 0loss/FRt1/assignment_loss‚0.7790343774934855 0loss/FRt2/assignment_loss‚2.0505200273850384 1loss/FBt2/assignment_loss‚0.11250244549862669 %loss/total_loss‚9.309712948748972 epoch‚14 trainer/global_step‚5699 _runtime‚3823.483815439 2loss/SRqqt2/assignment_loss‚0.6603520061359635ÕÜo}™H½‚(:½ œ„õҙ˃¿$ disk.nvme1n1.out‚11284.416015625( proc.memory.availableMB‚ 192.10546875 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚23221.07470703125# proc.memory.rssMB‚ 2971.65234375* proc.memory.percent‚18.137526512145996 cpu‚25.818655086757854 proc.cpu.threads‚17 disk.md0.out‚ 100.42578125 disk.dm-1.out‚ 3208.44140625 disk.rbd0.in‚0& disk.nvme1c1n1.out‚11284.416015625$ memory_percent‚98.82748126983643) disk./.usagePercent‚76.20144266197802% disk./.usageGB‚1169.8826484680176 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 network.sent‚ 8.378491e+06 network.recv‚ 1.045078e+06# disk.dm-0.out‚31080.94384765625( disk.nvme0c0n1.out‚23221.07470703125C1^Úž¾‚(:“ œ„õÒ®ôí¿ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚115.154' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.38466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýð.XD¿‚(:¹ «„õÒŒÿã¾) disk./.usagePercent‚76.18598559065414% disk./.usageGB‚1169.6453437805176 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚23221.69384765625# disk.nvme1n1.out‚11331.42578125$ memory_percent‚98.82638454437256 network.recv‚ 1.049691e+06( disk.nvme0c0n1.out‚23221.69384765625 disk.nvme1n1.in‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2971.65234375 proc.cpu.threads‚17 network.sent‚ 8.441963e+06 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 3208.64453125( proc.memory.availableMB‚ 192.28515625 disk.nvme0c0n1.in‚0% disk.nvme1c1n1.out‚11331.42578125 disk.md0.out‚ 100.640625# disk.dm-0.out‚31127.89697265625* proc.memory.percent‚18.137526512145996 cpu‚25.820596893679493Ç䣮“À‚(:ˆ «„õÒöã©¿ gpu.0.gpu‚91.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚220.968' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚73.656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚g:ÃKÁ‚(:À º„õÒè‰Þ¾$ memory_percent‚98.82636070251465' proc.memory.availableMB‚ 192.2890625# proc.memory.rssMB‚ 2971.65234375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚23322.47607421875 cpu‚25.82200388145379% disk./.usageGB‚1169.5811309814453 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11367.79638671875# disk.dm-0.out‚31264.05126953125* proc.memory.percent‚18.137526512145996 network.sent‚ 8.446745e+06 network.recv‚ 1.05305e+06( disk./.usagePercent‚76.1818030276141 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 101.02734375 disk.dm-1.in‚0& disk.nvme0n1.out‚23322.47607421875( disk.nvme1c1n1.out‚11367.79638671875 disk.dm-0.in‚0" disk.dm-1.out‚3208.80126953125¯6.‚(:’ º„õÒ¨ª¿ gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚222.905' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.30166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j-ŽT2Ã- _wandbruntime‚3855  _runtime‚3855áçÛÇÄÁ & lr-AdamW/pg1‚0.0001327269357823282 & lr-AdamW/pg2‚0.0001327269357823282 trainer/global_step‚5749 % _timestamp‚1.7844966992859282e+09 _runtime‚3855.887783703 _step‚470X¥ õÌÅÆ _runtime‚3855.887783703 _step‚470 & lr-AdamW/pg1‚0.0001327269357823282 & lr-AdamW/pg2‚0.0001327269357823282 trainer/global_step‚5749 % _timestamp‚1.7844966992859282e+09Öö´%‹2Æ- _wandbruntime‚3855  _runtime‚3855ªÎåæ'ÇàO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445546386961378 _runtime‚3855.888813875 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚5749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚450822594331136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450822488631808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚446080589628928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45575921 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 % _timestamp‚1.7844966992869806e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43606058 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4728993616204 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338966346263040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57840342 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57840342 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30588867 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13017187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334084306960896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57830199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334084401070592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57830199 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440817480131600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450822594331136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445546490787164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4741916042240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4881824137216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45575921 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚450822488631808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103406120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚446080572589568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4408174971709606€ð¸ë'ÈåO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45575923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13017012 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450822488631808 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚446080589628928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 121055232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45575921 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 % _timestamp‚1.7844966992869806e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43606058 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4728993616204 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338966346263040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57840342 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57840342 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30588867 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13017187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334084306960896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57830199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334084401070592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57830199 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215164928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440817480131600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450822594331136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚179 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445546490787164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4741916042240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4881824137216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45575921 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚450822488631808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103406120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚446080572589568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440817497170960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445546386961378 _runtime‚3855.888813875 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚5749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚450822594331136 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103433728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚450898424013824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4882638065152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57850176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103423569 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334140538798592 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45583550 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445621460981370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4882752821760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚472 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446155722255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 WCDeviceStatsMonitor.on_train_bat‹Ã€vûch_end/allocated_bytes.all.allocated‚450898529720320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13019405 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339023291620352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 trainer/global_step‚5749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103423569 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4742701757952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4742701757952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4742790425088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚446155722255872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚334140444688896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103433728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3856.599250484 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43613143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13019220 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4729777110916 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57850176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339023082754048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130¹òôÜõ&ËïM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4742701757952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚189 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4742790425088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚446155722255872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚185 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚334140444688896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103433728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3856.599250484 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43613143 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13019220 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4729777110916 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 208866304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57850176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339023082754048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57840019 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103433728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚450898424013824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4882638065152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57850176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103423569 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚334140538798592 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45583550 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445621460981370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4882752821760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114756608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚472 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚446155722255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚450898529720320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13019405 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339023291620352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 trainer/global_step‚5749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103423569 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4742701757952Øÿ£Ù2Ì- _wandbruntime‚3856  _runtime‚3856óå ̓ 2loss/SRqqt2/assignment_loss‚0.6169077052403517 1loss/SRqqt1/detection_loss‚0.6676769646211729 /loss/FBt2/detection_loss‚0.3666938088301501 trainer/global_step‚5749 % _timestamp‚1.7844966999982824e+09 0loss/FRt2/assignment_loss‚2.0282998793367026 1loss/SRqqt1/assignment_loss‚0.580985588970074 2loss/FBt1/assignment_loss‚0.004716660715712915 1loss/SRqqt2/detection_loss‚0.9548516101063222 %loss/total_loss‚8.541170078147067 /loss/FRt1/detection_loss‚1.4482688874300373 0loss/FBt1/detection_loss‚0.42378468095121763 _runtime‚3856.599828125 0loss/FRt1/assignment_loss‚0.7590932543654191 1loss/FBt2/assignment_loss‚0.13617742834621377 /loss/FRt2/detection_loss‚0.5537136092336922 epoch‚14 _step‚4739ètËΈ 0loss/FRt2/assignment_loss‚2.0282998793367026 1loss/SRqqt1/assignment_loss‚0.580985588970074 2loss/FBt1/assignment_loss‚0.004716660715712915 1loss/SRqqt2/detection_loss‚0.9548516101063222 %loss/total_loss‚8.541170078147067 /loss/FRt1/detection_loss‚1.4482688874300373 0loss/FBt1/detection_loss‚0.42378468095121763 _runtime‚3856.599828125 0loss/FRt1/assignment_loss‚0.7590932543654191 1loss/FBt2/assignment_loss‚0.13617742834621377 /loss/FRt2/detection_loss‚0.5537136092336922 epoch‚14 _step‚473 2loss/SRqqt2/assignment_loss‚0.6169077052403517 1loss/SRqqt1/detection_loss‚0.6676769646211729 /loss/FBt2/detection_loss‚0.3666938088301501 trainer/global_step‚5749 % _timestamp‚1.7844966999982824e+09Ù,)->NÏ‚(:à ɄõÒ£Ö„¿# disk.dm-0.out‚31321.34423828125" disk.dm-1.out‚3208.89501953125# proc.memory.rssMB‚ 2971.65234375* proc.memory.percent‚18.137526512145996) disk./.usagePercent‚76.18180377303612 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚98.82657527923584 cpu‚25.824038839899952 network.sent‚ 8.464366e+06& disk.nvme0n1.out‚23379.96923828125( disk.nvme1c1n1.out‚11368.22314453125 disk.nvme1n1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.056708e+06$ disk./.usageGB‚1169.581142425537( disk.nvme0c0n1.out‚23379.96923828125& disk.nvme1n1.out‚11368.22314453125 disk.dm-0.in‚0( proc.memory.availableMB‚ 192.25390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 101.27734375€é`ÜЂ(:‘ É„õÒ™ý¶¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.583' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.19433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àD[ÖLÑ‚(:Á Ø„õÒ›¿* proc.memory.percent‚18.137526512145996 cpu‚25.826252250822265 network.sent‚ 8.514682e+06 network.recv‚ 1.060895e+06) disk./.usagePercent‚76.18180253066606 disk.nvme0n1.in‚0& disk.nvme0n1.out‚23395.49462890625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 192.25390625% disk./.usageGB‚1169.5811233520508 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚23395.49462890625& disk.nvme1n1.out‚11370.51220703125 disk.md0.in‚0 disk.md0.out‚ 101.71875$ memory_percent‚98.82657527923584 disk.rbd0.out‚0( disk.nvme1c1n1.out‚11370.51220703125 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.dm-1.out‚3208.99755859375 disk.dm-0.in‚0# disk.dm-0.out‚31338.09716796875# proc.memory.rssMB‚ 2971.65234375òÒŸoÒ‚(:’ Ø„õÒ¥ÊÉ¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.728' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.242666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{®2Ó- _wandbruntime‚3888  _runtime‚3888Ö³øÈÔ _step‚474 ' lr-AdamW/pg1‚0.00013223910994306713 ' lr-AdamW/pg2‚0.00013223910994306713 trainer/global_step‚5799 % _timestamp‚1.7844967323078344e+09 _runtime‚ 3888.90964987æ ÃþÍÕÇ ' lr-AdamW/pg2‚0.00013223910994306713 trainer/global_step‚5799 % _timestamp‚1.7844967323078344e+09 _runtime‚ 3888.90964987 _step‚474 ' lr-AdamW/pg1‚0.00013223910994306713Ú…âÆ2Ö- _wandbruntime‚3888  _runtime‚3888j}Ìšå'×ßO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58323051 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4781265982464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58333194 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45957371 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4768145360586 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30841667 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43966922 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 _runtime‚3888.914329249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449300151761786 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336891193360896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43967109 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45957371 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104280422 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4922394354688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444532006401200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4781265982464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454619321887232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341813811269120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449838055904768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚475 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4922523798528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454619427586560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4781354642432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 223553536 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58333194 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13125438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58323051 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4781265982464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚187 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58333194 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45957371 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4768145360586 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30841667 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43966922 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 _runtime‚3888.914329249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449300151761786 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336891193360896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43967109 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45957371 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104280422 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4922394354688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444532006401200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4781265982464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454619321887232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341813811269120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129443840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449838055904768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Û°Eèo2Ù- _wandbruntime‚3889  _runtime‚3889M'†ð&ÚêM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444606313965952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4768928855186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45965000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104308030 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4782140365312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449913205571072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58343028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4923203039744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7844967327210708e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104297871 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45965000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚194 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58332871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚198 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45965002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30846723 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4769015641724 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449375225781778 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454695362975744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚449913222610432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 _runtime‚3889.323096881 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚449913205571072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43974072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30846727 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449375329607676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4782051698176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336947425198592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104308030 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚449913222610432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43974270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45965002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336947331088896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782051698176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58343028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454695362975744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13127349 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341870534128640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45965002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336947331088896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782051698176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58343028 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454695362975744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13127349 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341870534128640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104308030 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 223546368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4782140365312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449913205571072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58343028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4923203039744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7844967327210708e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104297871 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45965000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚194 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58332871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚198 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45965002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30846723 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4769015641724 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449375225781778 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454695362975744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚449913222610432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 _runtime‚3889.323096881 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚449913205571072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43974072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30846727 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449375329607676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4782051698176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336947425198592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812Ü,ÜC°2Ü- _wandbruntime‚3889  _runtime‚3889æÒïáÝ 0loss/FRt2/assignment_loss‚1.9997910716274911 trainer/global_step‚5799 0loss/FRt1/assignment_loss‚0.8162919655764774 2loss/SRqqt1/assignment_loss‚0.5984678866265114 2loss/SRqqt2/assignment_loss‚0.6005185925362942 2loss/FBt1/assignment_loss‚0.005774001746404381 /loss/FRt1/detection_loss‚1.6057116182408098 0loss/SRqqt2/detection_loss‚1.106160572049423 % _timestamp‚1.7844967327220867e+09 _step‚477 1loss/FBt2/assignment_loss‚0.08210699914148283 /loss/FRt2/detection_loss‚0.5665477176590162 0loss/SRqqt1/detection_loss‚0.694830582570885 0loss/FBt1/detection_loss‚0.49582428906591974 _runtime‚3889.323736822 epoch‚14 /loss/FBt2/detection_loss‚0.6502858558442535 $loss/total_loss‚9.22231115268497¯ox Þ† 0loss/FRt1/assignment_loss‚0.8162919655764774 2loss/SRqqt1/assignment_loss‚0.5984678866265114 2loss/SRqqt2/assignment_loss‚0.6005185925362942 2loss/FBt1/assignment_loss‚0.005774001746404381 /loss/FRt1/detection_loss‚1.6057116182408098 0loss/SRqqt2/detection_loss‚1.106160572049423 % _timestamp‚1.7844967327220867e+09 _step‚477 1loss/FBt2/assignment_loss‚0.08210699914148283 /loss/FRt2/detection_loss‚0.5665477176590162 0loss/SRqqt1/detection_loss‚0.694830582570885 0loss/FBt1/detection_loss‚0.49582428906591974 _runtime‚3889.323736822 epoch‚14 /loss/FBt2/detection_loss‚0.6502858558442535 $loss/total_loss‚9.22231115268497 0loss/FRt2/assignment_loss‚1.9997910716274911 trainer/global_step‚5799ÝðiLß‚(:Á ç„õÒঊ¿% proc.memory.availableMB‚ 192.21875 network.sent‚ 8.517705e+06 network.recv‚ 1.064196e+06 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚23396.16162109375 proc.cpu.threads‚17& disk.nvme0n1.out‚23396.16162109375 disk.md0.in‚0 disk.md0.out‚ 101.94140625 disk.nvme0c0n1.in‚0$ memory_percent‚98.82678985595703# proc.memory.rssMB‚ 2971.65234375* proc.memory.percent‚18.137526512145996 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚11370.91357421875" disk.dm-1.out‚3209.07568359375 disk.rbd0.in‚0 cpu‚25.828325264252115) disk./.usagePercent‚76.18179756118583% disk./.usageGB‚1169.5810470581055& disk.nvme1n1.out‚11370.91357421875 disk.dm-0.in‚0# disk.dm-0.out‚31338.58154296875 disk.dm-1.in‚0•w­0žà‚(:“ ç„õÒý¦ê¿ gpu.0.gpu‚56.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚128.206' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.73533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºó‹ÌJá‚(:¿ ö„õÒ溚¿ disk.dm-1.out‚ 3209.17578125 disk.rbd0.out‚0 disk.nvme0n1.in‚0* proc.memory.percent‚18.137526512145996) disk./.usagePercent‚76.18536564799477% disk./.usageGB‚1169.6358261108398 disk.md0.out‚ 102.27734375 disk.nvme0c0n1.in‚0( disk.nvme0c0n1.out‚23407.84716796875$ memory_percent‚98.82676601409912# proc.memory.rssMB‚ 2971.65234375 proc.cpu.threads‚17' disk.nvme1c1n1.out‚11372.9833984375 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚23407.84716796875( proc.memory.availableMB‚ 192.22265625 cpu‚25.830569695192963 network.sent‚ 8.581987e+06% disk.nvme1n1.out‚11372.9833984375 disk.md0.in‚0 disk.dm-0.in‚0 network.recv‚ 1.068301e+06 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚31351.50048828125ôªþšžâ‚(:“ ö„õÒ šÎ¿ gpu.0.gpu‚76.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚227.627' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.87566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰¾ìñ2ã- _wandbruntime‚3922  _runtime‚3922(¨Éäà ' lr-AdamW/pg1‚0.00013174794334236748 ' lr-AdamW/pg2‚0.00013174794334236748 trainer/global_step‚5849 % _timestamp‚1.7844967654440205e+09 _runtime‚3922.046054919 _step‚478ýMùÎåÈ ' lr-AdamW/pg1‚0.00013174794334236748 ' lr-AdamW/pg2‚0.00013174794334236748 trainer/global_step‚5849 % _timestamp‚1.7844967654440205e+09 _runtime‚3922.046054919 _step‚478Þ'G.2æ- _wandbruntime‚3922  _runtime‚3922w$ô*çàO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105164869 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58826046 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 trainer/global_step‚5849 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonito•ÿËͼ%r.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44326262 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453053916562194 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13231795 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458416155142656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339698173870592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448246532670800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4820704582656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46338823 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4807470677820 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458416260841984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339698079760896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453595539219968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚3922.046670211 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13231974 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46338821 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 _step‚479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44326445 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453595539219968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200Ië'èåO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458416155142656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚448246549710160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4820704582656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4820615922688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58815903 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31094467 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458416260841984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344661009838592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4807383891394 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31094471 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453054020387980 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46338821 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344661227100672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4963053230080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453595556259328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚682 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458416155142656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339698173870592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448246532670800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4820704582656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46338823 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4807470677820 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458416260841984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339698079760896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453595539219968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 _runtime‚3922.046670211 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13231974 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46338821 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 123152384 _step‚479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44326445 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚453595539219968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105164869 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58826046 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚179 trainer/global_step‚5849 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44326262 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453053916562194 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13231795 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46346452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4821490305536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4821401638400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339754217488896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58835880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 _runtime‚3922.460580477 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182332 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58825723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458492196231168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13233973 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31099527 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58825723 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31099523 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458492090524672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46346450 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44333496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13234145 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105172173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344718176021504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453129094408084 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4963741277696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453128990582186 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4963864422912 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453670705925632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448320840235552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚5849 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4821401638400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182332 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4808254172532 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453670688886272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58835880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4808167385994 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216dBÐ ëîM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46346452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44333672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105172173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344718176021504 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453129094408084 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4963741277696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453128990582186 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4963864422912 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453670705925632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448320840235552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 trainer/global_step‚5849 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4821401638400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182332 K;De‹0vßviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4808254172532 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453670688886272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58835880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4808167385994 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 123145216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.784496765858542e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105172173 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458492196231168 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46346452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4821490305536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4821401638400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339754217488896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58835880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 _runtime‚3922.460580477 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 14365491200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182332 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58825723 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458492196231168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13233973 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31099527 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58825723 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31099523 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458492090524672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46346450 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44333496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13234145 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 F‚2ö- _wandbruntime‚3953  _runtime‚3953{?·æ'÷àO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46720271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44687025 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59318898 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4846622422202 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4846709208628 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚5899 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31347267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342504966160896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106039171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚3953.111654798 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 _step‚483 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46720273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31347271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13339939 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13339758 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860054522880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451961058940400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457353022535168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342505060270592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462212988398080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44687210 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106029026 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4860054522880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347508403101696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456807785188388 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457353039574528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696Â_›Ië'øåO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46720273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31347271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13339939 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13339758 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860054522880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451961058940400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚457353022535168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342505060270592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462212988398080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44687210 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106029026 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4860054522880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347508403101696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456807785188388 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457353039574528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46720271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44687025 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59318898 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221456384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4846622422202 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4846709208628 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 trainer/global_step‚5899 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31347267 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚821 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342504966160896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106039171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 _runtime‚3953.111654798 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127346688 _step‚483ãÂL³92ù- _wandbruntime‚3953  _runtime‚3953… 7®úéM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59328732 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462289029486592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46727900 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonh®ÌA%itor.on_train_batch_end/segment.all.allocated‚812 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚457428172201472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59318575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4847405916802 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59328732 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚457428189240832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457428189240832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _runtime‚3953.504757006 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4847492703340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59318575 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860751578624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347565576296960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784496796902768e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106046475 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452035349465792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106056634 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44694130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456882859208492 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342561103888896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452035366505152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13341993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4860840245760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342561197998592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46727902 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4860840245760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44694320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚5899 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347565350653440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457428172201472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5004246764544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FRô&ûîM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347565350653440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457428172201472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5004246764544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚457428172201472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59318575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4847405916802 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59328732 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚457428189240832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457428189240832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131533824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 _runtime‚3953.504757006 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4847492703340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14638120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59318575 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860751578624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347565576296960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784496796902768e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106046475 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452035349465792 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106056634 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44694130 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456882859208492 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342561103888896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452035366505152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13341993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4860840245760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342561197998592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚682 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46727902 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4860840245760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44694320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚190 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 trainer/global_step‚5899ä^Õæ2ü- _wandbruntime‚3953  _runtime‚3953xrIý 0loss/FRt1/assignment_loss‚0.7944029982456428 1loss/SRqqt2/detection_loss‚1.1492204015440721 0loss/FRt2/assignment_loss‚1.9535304802594726 /loss/SRqqt1/assignment_loss‚0.5821287336365 2loss/SRqqt2/assignment_loss‚0.6595249270533073 2loss/FBt1/assignment_loss‚0.017350540413818373 %loss/total_loss‚8.968158878579214 _runtime‚3953.505255907 _step‚485 /loss/FRt1/detection_loss‚1.3927177581285497 1loss/SRqqt1/detection_loss‚0.6532032617474063 trainer/global_step‚5899 2loss/FBt2/assignment_loss‚0.057149481832754034 /loss/FRt2/detection_loss‚0.5633060169307222 /loss/FBt1/detection_loss‚0.5068032624107515 /loss/FBt2/detection_loss‚0.6388210163762172 epoch‚14 % _timestamp‚1.7844967969036956e+09ùÏØŠ þ† 2loss/SRqqt2/assignment_loss‚0.6595249270533073 2loss/FBt1/assignment_loss‚0.017350540413818373 %loss/total_loss‚8.968158878579214 _runtime‚3953.505255907 _step‚485 /loss/FRt1/detection_loss‚1.3927177581285497 1loss/SRqqt1/detection_loss‚0.6532032617474063 trainer/global_step‚5899 2loss/FBt2/assignment_loss‚0.057149481832754034 /loss/FRt2/detection_loss‚0.5633060169307222 /loss/FBt1/detection_loss‚0.5068032624107515 /loss/FBt2/detection_loss‚0.6388210163762172 epoch‚14 % _timestamp‚1.7844967969036956e+09 0loss/FRt1/assignment_loss‚0.7944029982456428 1loss/SRqqt2/detection_loss‚1.1492204015440721 0loss/FRt2/assignment_loss‚1.9535304802594726 /loss/SRqqt1/assignment_loss‚0.5821287336365å Ü\&Eÿ‚(:º £…õÒÎÚ•¿) disk./.usagePercent‚76.18180352456211% disk./.usageGB‚1169.5811386108398 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.079519e+06 disk.dm-1.out‚ 3209.578125$ memory_percent‚98.82786273956299( proc.memory.availableMB‚ 192.04296875 cpu‚25.835445619706835 proc.cpu.threads‚17& disk.nvme0c0n1.out‚23586.869140625 disk.md0.out‚ 103.3515625# proc.memory.rssMB‚ 2971.65234375* proc.memory.percent‚18.137526512145996 network.sent‚ 8.664876e+06 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚23586.869140625( disk.nvme1c1n1.out‚11448.36572265625& disk.nvme1n1.out‚11448.36572265625 disk.md0.in‚0 disk.dm-0.in‚0# disk.dm-0.out‚31603.17529296875CܱIž€‚(:“ £…õÒä¼Ì¿ gpu.0.gpu‚89.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚50.0 gpu.0.powerWatts‚116.234' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.74466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù ˜¡L‚(:Á ²…õÒžþß¾) disk./.usagePercent‚76.20086620227072( disk.nvme1c1n1.out‚11452.51708984375 cpu‚25.834628605825905 network.recv‚ 1.084063e+06 disk.md0.out‚ 103.7734375 disk.dm-0.in‚0( disk.nvme0c0n1.out‚23680.51708984375 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚98.82898330688477& proc.memory.availableMB‚ 191.859375 network.sent‚ 8.716619e+06# disk.dm-0.out‚31699.87158203125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2971.65234375* proc.memory.percent‚18.137526512145996% disk./.usageGB‚1169.8737983703613 disk.md0.in‚0" disk.dm-1.out‚3209.76220703125 disk.nvme0c0n1.in‚0& disk.nvme0n1.out‚23680.51708984375& disk.nvme1n1.out‚11452.51708984375 proc.cpu.threads‚17õvÚ¥œ‚‚(:‘ ²…õÒû—¿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚49.0 gpu.0.powerWatts‚188.797' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.93233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p%ò2ƒ- _wandbruntime‚3984  _runtime‚3984t|É„Ã _step‚486 ' lr-AdamW/pg1‚0.00013075571425604887 ' lr-AdamW/pg2‚0.00013075571425604887 trainer/global_step‚5949 % _timestamp‚1.7844968275197012e+09 _runtime‚3984.123285848=GÃÎ…È ' lr-AdamW/pg1‚0.00013075571425604887 ' lr-AdamW/pg2‚0.00013075571425604887 trainer/global_step‚5949 % _timestamp‚1.7844968275197012e+09 _runtime‚3984.123285848 _step‚486æ켩-2†- _wandbruntime‚3984  _runtime‚3984ùlKk^!‡àO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466009927352832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13446151 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 225650688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31600071 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455675585210000 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 % _timestamp‚1.7844968275202937e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47101721 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45046222 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106913473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59801607 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460561446163010 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106903328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461110505850368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4899315803136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31600067 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350355978819584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466009821653504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47101723 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461110522889728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460561549988796 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59801607 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4885947739436 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4899404463104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59811750 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45046037 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350355753168896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345311852560896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_poœJÚ؈ol.freed‚ 4885860953010 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460561446163010 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106903328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚181 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461110505850368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4899315803136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31600067 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350355978819584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466009821653504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 272629760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47101723 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚461110522889728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚460561549988796 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 14365491200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59801607 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4885947739436 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4899404463104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59811750 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45046037 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350355753168896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345311852560896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4885860953010 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31600071 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455675585210000 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚812 % _timestamp‚1.7844968275202937e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47101721 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45046222 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106913473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248ç $–2‰- _wandbruntime‚3984  _runtime‚3984R`­êð&ŠêM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12301877248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4900101518848 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59811427 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚5949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚466085862742016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231934976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45053348 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350412925331968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460636520183002 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31605123 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45053162 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47109350 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461185655516672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31605127 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460636624008900 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47109350 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13448039 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7844968279204996e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345368084398592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59821584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106930936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47109352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106920777 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345367990288896 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461185672556032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455749875735392 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚466085862742016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4886731234148 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350412693396992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137825280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59811427 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5044703108096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4900190185984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4900101518848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47109352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4900190185984 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚466085862742016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231934976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45053348 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350412925331968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460636520183002 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31605123 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45053162 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2342719488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47109350 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚461185655516672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31605127 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460636624008900 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47109350 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13448039 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 % _timestamp‚1.7844968279204996e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345368084398592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59821584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12065026048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106930936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47109352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106920777 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345367990288896 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461185672556032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455749875735392 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚488 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚466085862742016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4886731234148 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350412693396992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137825280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59811427 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5044703108096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4900190185984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4900101518848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47109352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4900190185984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59811427 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 trainer/global_step‚5949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459390128518960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45405215 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60294459 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5084525722624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5084392084480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47483173 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348118738960896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31852871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60294459 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13552344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13552162 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464868006204928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464867989165568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4938754403328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469806654908928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469806654908928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31852867 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4925186270244 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227747840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348118833070592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464315210963418 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4938754403328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464867989165568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47483171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353203358793216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459390111479600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469806760608256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107787775 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47483171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0*´æë'˜åO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4938665743360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚682 _runtime‚4016.739913945 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚812 trainer/global_step‚5999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107777630 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469806654908928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 269300736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31852867 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88659968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103825786 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4925186270244 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227747840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14638120960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12065026048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348118833070592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 14365491200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12065026048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464315210963418 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚812 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4938754403328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105699328 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464867989165568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47483171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353203358793216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459390111479600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469806760608256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12301877248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107787775 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47483171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12301877248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚182 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 272629760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚469806760608256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60304602 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353203131045376 % _timestamp‚1.7844968601380603e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105699328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464868006204928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464315314789204 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1509 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459390128518960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 272629760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45405215 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2342719488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133638144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 14365491200 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60294459 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚186 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5084525722624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5084392084480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11862167552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86786426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14638120960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88659968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47483173 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348118738960896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 267417279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31852871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60294459 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13552344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13552162 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2372707840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚812 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464868006204928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464867989165568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12106884175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚94109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4938754403328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469806654908928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 269300736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150006272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14638120960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚350 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚5999 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464390388809308 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5085335612928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47490802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459464402004992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464390284983410 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47490802 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5085204079104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4925882978418 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353260306411520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31857923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60314436 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464943155871232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60304279 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353260080768000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 % _timestamp‚1.7844968605521164e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚469882695997440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464943138831872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4939540126208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4939540126208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚4017.154240771 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47490800 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45412270 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 _step‚492 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107795079 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31857927 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107805238 cODeviceStatsMon¤k® itor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348174970798592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4939451459072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348174876688896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459464419044352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13554347\`äõ&›ïM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 269300736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4939451459072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348174876688896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 272629760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459464419044352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 267417279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88667136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13554347 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚94109696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107805238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150006272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4925969764956 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚812 trainer/global_step‚5999 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464390388809308 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2372707840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5085335612928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86786538 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14638120960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚350 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105706496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47490802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1509 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459464402004992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚130 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464390284983410 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47490802 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12301877248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 269300736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5085204079104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4925882978418 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 272629760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353260306411520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 272629760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31857923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225643520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚812 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60314436 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464943155871232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60304279 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚821 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353260080768000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103825898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚130 % _timestamp‚1.7844968605521164e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚469882695997440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464943138831872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4939540126208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4939540126208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88667136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 14365491200 _runtime‚4017.154240771 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47490800 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45412270 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚812 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11862167552 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12106884175 _step‚492 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107795079 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12065026048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14638120960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31857927 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105706496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107805238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348174970798592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 14365491200ìa¡®Æ2œ- _wandbruntime‚4017  _runtime‚40177Í°ï€ /loss/FRt2/detection_loss‚0.5593122910180914 1loss/SRqqt1/detection_loss‚0.6481736427917758 0loss/FRt2/assignment_loss‚1.8949738537407694 2loss/SRqqt1/assignment_loss‚0.5394353333202012 1loss/FBt1/assignment_loss‚0.03719470287780349 1loss/FBt2/assignment_loss‚0.10985884165176624 /loss/FBt1/detection_loss‚0.5992260098996671 /loss/FBt2/detection_loss‚0.9874545089674719 %loss/total_loss‚9.113807158896542 # _timestamp‚1.78449686055337e+09 /loss/FRt1/detection_loss‚1.5010477997737781 1loss/SRqqt2/detection_loss‚0.8853748355759308 epoch‚14 trainer/global_step‚5999 _runtime‚4017.154985332 _step‚493 0loss/FRt1/assignment_loss‚0.7430150078134499 2loss/SRqqt2/assignment_loss‚0.60874033146583682 ž… 0loss/FRt2/assignment_loss‚1.8949738537407694 2loss/SRqqt1/assignment_loss‚0.5394353333202012 1loss/FBt1/assignment_loss‚0.03719470287780349 1loss/FBt2/assignment_loss‚0.10985884165176624 /loss/FBt1/detection_loss‚0.5992260098996671 /loss/FBt2/detection_loss‚0.9874545089674719 %loss/total_loss‚9.113807158896542 # _timestamp‚1.78449686055337e+09 /loss/FRt1/detection_loss‚1.5010477997737781 1loss/SRqqt2/detection_loss‚0.8853748355759308 epoch‚14 trainer/global_step‚5999 _runtime‚4017.154985332 _step‚493 0loss/FRt1/assignment_loss‚0.7430150078134499 2loss/SRqqt2/assignment_loss‚0.6087403314658368 /loss/FRt2/detection_loss‚0.5593122910180914 1loss/SRqqt1/detection_loss‚0.6481736427917758í8°•IŸ‚(:¾ ß…õÒåÂø¿ disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.095043e+06 disk.dm-1.out‚3210.052734375( disk.nvme0c0n1.out‚23814.37451171875 disk.nvme0n1.in‚0& disk.nvme0n1.out‚23814.37451171875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚11517.8662109375) proc.memory.percent‚18.14558506011963 disk.md0.in‚0 disk.md0.out‚ 104.75390625# disk.dm-0.out‚31896.67822265625 disk.rbd0.out‚0% disk.nvme1n1.out‚11517.8662109375# proc.memory.rssMB‚ 2972.97265625 cpu‚25.838688668281286 network.sent‚ 8.785862e+06) disk./.usagePercent‚76.00472255670138% disk./.usageGB‚1166.8625030517578 disk.nvme0c0n1.in‚0# memory_percent‚99.2293119430542( proc.memory.availableMB‚ 126.26953125F.+ ‚(:’ ß…õÒéÜ´À gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.681655883789062* gpu.0.memoryAllocatedBytes‚ 15188426752 gpu.0.temp‚48.0 gpu.0.powerWatts‚114.472' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.157333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:láq2¡- _wandbruntime‚4031  _runtime‚4031eE>áí=¢ç{ :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20802171528339386 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5839877724647522 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2160407304763794 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3337041437625885 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025117043405771255 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5220833420753479 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6556634902954102 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907164573669434 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1480223834514618 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 (particle/accuracy‚0.8486754298210144 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6181467771530151 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9217567443847656 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 'particle/f_score‚0.6405962705612183 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7972410917282104 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29519495368003845 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6721615195274353 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22867873311042786 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 +jet/accuracy_3_of_3‚0.07316494733095169 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22760377824306488 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6721615195274353 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6545295715332031 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997080385684967 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943540692329407 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03303533047437668 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8383219838142395 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18553176522254944 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6161506175994873 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19535546004772186 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16099773347377777 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18431302905082703 )loss/val_total_loss‚9.054805224890847 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9785394072532654 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7103136777877808 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43901747465133667 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 /particle/accuracy_3_of_4‚0.9910430908203125 trainer/global_step‚5999 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8453514575958252 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930174946784973 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_2‚0.7717479467391968 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02126540243625641 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.681014895439148 +jet/accuracy_2_of_3‚0.45502522587776184 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6664468050003052 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11785627901554108 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.042124729603528976 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3238416314125061 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6334162354469299 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6962445378303528 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976538419723511 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7284202575683594 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6108221411705017 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_1‚0.3243366479873657 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9956639409065247 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 .particle/accuracy_4_of_4‚0.941148579120636 _runtime‚4031.266220489 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6567700505256653 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015460851602256298 % _timestamp‚1.7844968746635542e+09 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2160864621400833 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9277394413948059 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6938880085945129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14521172642707825 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6992396712303162 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3093400001525879 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5549065470695496 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.837660551071167 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.551858127117157 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6999768614768982 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1681453138589859 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9947409629821777 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19659173488616943 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 +jet/accuracy_2_of_2‚0.13113196194171906 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1825244277715683 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967843294143677 +particle/specificity‚0.9090224504470825 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979884624481201 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8598235249519348 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚14 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951315522193909 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6700500249862671 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5756007432937622 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37082377076148987 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44704610109329224 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6510671973228455 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5989583730697632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47682851552963257 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13459396362304688 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.042526617646217346 *jet/accuracy_1_of_4‚0.9551719427108765 *jet/accuracy_2_of_4‚0.5481659770011902 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.272077739238739 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15517300367355347 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6379343271255493 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5440154075622559 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5934309959411621 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6318175792694092 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8353930115699768 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6468194723129272 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.028933173045516014 particle/accuracy_1_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.373424768447876 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45793724060058594 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5786692500114441 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6379343271255493 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6591607332229614 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12784788012504578 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23468832671642303 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7401917576789856 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9937176704406738 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03902778774499893 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_3‚0.9007691740989685 /particle/accuracy_2_of_4‚0.9968253374099731 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9863810539245605 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2160407304763794 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22796687483787537 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6063084602355957 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04395173862576485 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6275848150253296 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047678157687187195 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7080370187759399 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6231579780578613 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_4_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7350583076477051 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17457926273345947 /particle/accuracy_3_of_3‚0.9994933605194092 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5387508273124695 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6029242277145386 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9892697334289551 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17500001192092896 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7356129884719849 7validation_average_jet_accuracy‚0.36144012212753296 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8944075703620911 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8680499196052551 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967843294143677 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11176107078790665 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21478259563446045 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863163232803345 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 +particle/sensitivity‚0.6280059814453125 _step‚494 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6040153503417969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17169302701950073 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6226122379302979 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.038334015756845474 *jet/accuracy_3_of_4‚0.0934242308139801 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2785407304763794 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04395173862576485o“¼t £ì{ 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44704610109329224 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6510671973228455 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5989583730697632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47682851552963257 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13459396362304688 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.042526617646217346 *jet/accuracy_1_of_4‚0.9551719427108765 *jet/accuracy_2_of_4‚0.5481659770011902 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.272077739238739 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15517300367355347 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6379343271255493 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5440154075622559 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5934309959411621 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6318175792694092 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8353930115699768 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6468194723129272 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.028933173045516014 particle/accuracy_1_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.373424768447876 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45793724060058594 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5786692500114441 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6379343271255493 particle/accuracy_5_of_5‚